neu_mark.explore

Looking at annotation strings before writing rules to parse them.

Every function takes a DataFrame and returns one; nothing prints, writes or needs DVID, so this is meant for a notebook: look at the vocabulary, write a rule, see what it missed, repeat. (Named explore so it does not shadow the standard library’s inspect.)

rules is a {name: callable} mapping. Each callable takes one row, a pandas Series, and returns a scalar, a sequence (for multi-valued facets), or None / an empty sequence when it did not fire. Read columns as r["instance"], not r.instance: a column named name or size would return the Series attribute instead. A Rule is such a callable, and RuleSet.coverage and unparsed() wrap the functions here.

Counts are by body, not by distinct string, so a vocabulary decision is weighted by how much data it touches.

Attributes

RELATIONS

Functions

apply_rules(frame, rules, *[, key]) → pandas.DataFrame	Run every rule over every row. One column per rule, values as tuples.
compare(frame, field, against, *, column[, key]) → pandas.DataFrame	Row-by-row: how a curated field relates to what a rule derives.
coverage(frame, rules, *[, top]) → pandas.DataFrame	Per-rule: how often it fired, how many distinct values, and the commonest.
instances(frame, column) → pandas.DataFrame	Distinct strings and how many bodies carry each, most common first. The first
near(token_frame, target, *[, cutoff, include_exact]) → pandas.DataFrame	Tokens close enough to target to be misspellings of it, with their weight.
normalize(value) → str | None	Strip whitespace and trailing doubt marks; None for anything missing,
split_tokens(value) → list[str]	One string's tokens, normalized, for use in rule bodies:
tokens(frame, column, *[, position, drop]) → pandas.DataFrame	Token frequency, for hunting controlled vocabulary and its misspellings.
unparsed(frame, rules, rule, *, column) → pandas.DataFrame	The strings a given rule did not fire on, ranked by how many bodies they cover.
variants(frame, column) → pandas.DataFrame	Raw strings that normalize() repaired, and what they became.

Module Contents

neu_mark.explore.apply_rules(frame: pandas.DataFrame, rules: Mapping[str, Callable], *, key: str = 'body') → pandas.DataFrame

Run every rule over every row. One column per rule, values as tuples.

An empty tuple means the rule did not fire. A rule that raises is an error: it should return None for a value it cannot parse.

neu_mark.explore.compare(frame: pandas.DataFrame, field: str, against: Callable, *, column: str, key: str = 'body') → pandas.DataFrame

Row-by-row: how a curated field relates to what a rule derives.

One row per body with a relation from RELATIONS; summarise with .relation.value_counts(). A disagreement is not necessarily an error: type is often the coarser label (LMC for an instance of L1/L3_D4(R)).

neu_mark.explore.coverage(frame: pandas.DataFrame, rules: Mapping[str, Callable], *, top: int = 3) → pandas.DataFrame

Per-rule: how often it fired, how many distinct values, and the commonest.

A rule that under-fires on a messy vocabulary looks the same as data that lacks the facet; coverage is how to tell.

neu_mark.explore.instances(frame: pandas.DataFrame, column: str) → pandas.DataFrame

Distinct strings and how many bodies carry each, most common first. The first thing to look at: typically a few thousand strings, a reviewable vocabulary.

neu_mark.explore.near(token_frame: pandas.DataFrame, target: str, *, cutoff: float = 0.72, include_exact: bool = False) → pandas.DataFrame

Tokens close enough to target to be misspellings of it, with their weight.

Uses difflib similarity, since real misspellings cannot be listed in advance.

neu_mark.explore.normalize(value: Any) → str | None

Strip whitespace and trailing doubt marks; None for anything missing, including pd.NA (which would otherwise become the string "<NA>").

neu_mark.explore.split_tokens(value: Any) → list[str]

One string’s tokens, normalized, for use in rule bodies: "LN_C5(L)_NCL" -> ['LN', 'C5', 'L', 'NCL']; missing values give [].

neu_mark.explore.tokens(frame: pandas.DataFrame, column: str, *, position: str = 'trailing', drop: str | None = None) → pandas.DataFrame

Token frequency, for hunting controlled vocabulary and its misspellings.

position is "trailing" (the last token, where completeness markers and flags live), "leading", or "all". A string with no separator is its own token.

drop is a regex removed before tokenizing, for a facet already accounted for; since a parenthesized side makes L/R the trailing token of most strings, drop=r"\((L|R)\)" is the usual first move.

Returns token, bodies and strings (how many distinct strings contain it).

neu_mark.explore.unparsed(frame: pandas.DataFrame, rules: Mapping[str, Callable], rule: str, *, column: str) → pandas.DataFrame

The strings a given rule did not fire on, ranked by how many bodies they cover.

The top of this list is the next rule to write, or the missing vocabulary entry.

neu_mark.explore.variants(frame: pandas.DataFrame, column: str) → pandas.DataFrame

Raw strings that normalize() repaired, and what they became.

Whitespace and trailing ? are repaired before tokens() and near() see a string, so this is where they show up. A trailing ? is an annotator’s doubt, worth watching. near() finds genuine misspellings (fragmnet).

Returns raw, normalized, bodies for every string that changed.

neu_mark.explore.RELATIONS = ('exact', 'case_only', 'derived_is_longer', 'field_is_longer', 'substring', 'unrelated',...