neu_mark.dvid¶

Fetching annotations out of DVID, for a given set of bodies.

Addressing and version resolution come from neu_vol.dvid: a dvid://server/uuid/instance URL, resolved once to a concrete node that the provenance record names.

Point annotations are read per body from /label/<body>, which exists only when the annotation instance is synced to a labelmap; that is checked up front. neuclease’s fetch_elements_for_bodies is not used, because it cannot return relationships. Bodies are fetched on a thread pool (the work is I/O bound), each with retry, and a body that still fails is recorded rather than raised. The default of 8 threads is a courtesy to a shared server, which answers overload with 503.

fetch_all_points() reads the whole instance by region instead, with each element’s body from the synced labelmap, so every partner resolves.

Attributes¶

`BODY_INSTANCE`
`COUNT_INSTANCE`
`DEFAULT_KEY_BOUNDARIES`
`DEFAULT_THREADS`
`MAX_SPLIT_DEPTH`
`ON_OVERLAP`
`POINT_INSTANCE`
`ROI_UNSPECIFIED`
`logger`

Functions¶

`fetch_all_body_annotations`(source, *[, boundaries]) → dict[str, Any]	Every record in a keyvalue instance, fetched as a series of bounded key ranges.
`fetch_all_points`(source, *[, threads]) → dict[str, Any]	Every element of the instance, by region, with each one's body from the labelmap.
`fetch_body_annotations`(source, bodies, *[, batch_size]) → dict[str, Any]	Fetch the per-body records from a keyvalue instance.
`fetch_points`(source, bodies, *[, threads, progress_every]) → dict[str, Any]	Fetch every element of every body, and join relationships to bodies.
`fetch_synapse_counts`(source, *[, min_total, min_pre, ...])	Bodies with at least this many synapses, with their pre/post counts.
`fetch_synapse_counts_for`(source, bodies)	Exact `pre`/`post`/`syn` for specific bodies, ignoring any threshold.
`fetch_voxel_counts`(source, bodies, *[, batch_size]) → dict[int, int]	`{body: voxels}` at full resolution, from the labelmap's own `/sizes`.
`key_ranges`([boundaries]) → list[tuple[str, str]]	Consecutive pairs of `boundaries` as `(key1, key2)` request bounds.
`label_point_rois`(source, points, rois, *[, ...]) → dict[str, Any]	Add a `roi` column to a points frame: which neuropil each synapse falls in.
`node_record`(source, **extra) → dict[str, Any]	The provenance record for this source, with whatever the caller wants added.
`open_bodies_source`(src, *[, prefer_locked]) → dict[str, Any]	Resolve and validate a body-annotation (keyvalue) source.
`open_counts_source`(src, *[, prefer_locked]) → dict[str, Any]	Resolve a synapse-count source: a `labelsz` index, or the instance it indexes.
`open_points_source`(src, *[, prefer_locked]) → dict[str, Any]	Resolve and validate a point-annotation source. Call this before anything else.
`open_source`(spec, *, expect[, prefer_locked]) → dict[str, Any]	Resolve the ref, check the instance type, and return the pinned spec.
`refine`(lo, hi) → list[str]	Boundaries subdividing `[lo, hi]` by appending one digit to `lo`.
`require_sync`(source) → list[str]	The labelmap(s) an annotation instance is synced to, or raise explaining why not.
`resolve_labelsz`(source) → dict[str, Any]	Point a source at the `labelsz` instance to query, from whatever was given.

Module Contents¶

neu_mark.dvid.fetch_all_body_annotations(source: Mapping[str, Any], *, boundaries: Sequence[str] = DEFAULT_KEY_BOUNDARIES) → dict[str, Any]¶

Every record in a keyvalue instance, fetched as a series of bounded key ranges.

Never calls /keys, which on a large instance times out at DVID’s proxy (504). The instance is read as keyrangevalues over ranges that cover the whole key space, so no total is needed, and values cost little more than keys. A range that fails is split (refine()) up to MAX_SPLIT_DEPTH times. Boundary keys arrive twice, since DVID’s bounds are inclusive; a repeat with a different value raises, as the snapshot would not be coherent.

neu_mark.dvid.fetch_all_points(source: Mapping[str, Any], *, threads: int = DEFAULT_THREADS) → dict[str, Any]¶

Every element of the instance, by region, with each one’s body from the labelmap.

The same tables as fetch_points(), but complete: both ends of every relationship were fetched, so every partner resolves. Elements come from neuclease.dvid.annotation.fetch_synapses_in_batches (DVID’s /blocks, in boxes, threads processes) and bodies from fetch_bodies_for_many_points on the synced labelmap at the same node (supervoxel per point, then DVID’s whole supervoxel->body map). An element on background has body 0, which connections treats as no body.

neu_mark.dvid.fetch_body_annotations(source: Mapping[str, Any], bodies: Sequence[int], *, batch_size: int = 10000) → dict[str, Any]¶

Fetch the per-body records from a keyvalue instance.

Uses fetch_keyvalues rather than neuclease’s fetch_body_annotations, which raises on any status outside a fixed vocabulary; statuses stay plain strings. Bodies with no record are absent from the result and listed in missing.

neu_mark.dvid.fetch_points(source: Mapping[str, Any], bodies: Sequence[int], *, threads: int = DEFAULT_THREADS, progress_every: int = 500) → dict[str, Any]¶

Fetch every element of every body, and join relationships to bodies.

Returns {"points", "relationships", "connections", "match", "failures", "elements"}. failures is {body: message} for bodies that still failed after retries; the run continues without them.

neu_mark.dvid.fetch_synapse_counts(source: Mapping[str, Any], *, min_total: int = 10, min_pre: int = 0, min_post: int = 0, limit: int | None = None)¶

Bodies with at least this many synapses, with their pre/post counts.

One ranked threshold query on AllSyn at the largest of the three minimums (AllSyn counts every type, so it is at least each one), then exact pre and post counts for the bodies it returns.

Returns a DataFrame indexed 0..n with body, pre, post, syn, ranked by syn descending.

neu_mark.dvid.fetch_synapse_counts_for(source: Mapping[str, Any], bodies: Sequence[int])¶

Exact pre/post/syn for specific bodies, ignoring any threshold.

fetch_synapse_counts() finds the bodies above a threshold; this counts a given list.

neu_mark.dvid.fetch_voxel_counts(source: Mapping[str, Any], bodies: Sequence[int], *, batch_size: int = 1000) → dict[int, int]¶

{body: voxels} at full resolution, from the labelmap’s own /sizes.

Not from labelsz, whose 'Voxels' type returns an empty result, not an error, without a labelvol sync. About a minute for 20,000 bodies.

neu_mark.dvid.key_ranges(boundaries: Sequence[str] = DEFAULT_KEY_BOUNDARIES) → list[tuple[str, str]]¶

Consecutive pairs of boundaries as (key1, key2) request bounds.

DVID’s key ranges are inclusive at both ends, so consecutive ranges overlap by one key where a key equals a boundary (body 3 is in both '2'..'3' and '3'..'4'). Collecting into a dict de-duplicates, but summing per-range counts overcounts.

neu_mark.dvid.label_point_rois(source: Mapping[str, Any], points, rois: Sequence[str], *, on_overlap: str = 'warn', processes: int = 0) → dict[str, Any]¶

Add a roi column to a points frame: which neuropil each synapse falls in.

Wraps neuclease.dvid.roi.determine_point_rois, which builds one combined label volume from the ROIs at scale 5 and samples it at each point’s scale-0 x/y/z columns.

Overlapping ROIs: the combined volume is built by writing each ROI in turn, so in an intersection the ROI passed last wins. With on_overlap="warn" the volume is unpacked again in reverse order (the fetch is not repeated), and the result reports how many synapses would be attributed differently (ambiguous, ambiguous_pairs). on_overlap="error" refuses instead; some datasets have subtracted variants such as INP(-ATL)(L) for a strict partition.

neu_mark.dvid.node_record(source: Mapping[str, Any], **extra: Any) → dict[str, Any]¶: The provenance record for this source, with whatever the caller wants added.

neu_mark.dvid.open_bodies_source(src: Mapping[str, Any], *, prefer_locked: bool = False) → dict[str, Any]¶: Resolve and validate a body-annotation (keyvalue) source.

neu_mark.dvid.open_counts_source(src: Mapping[str, Any], *, prefer_locked: bool = False) → dict[str, Any]¶: Resolve a synapse-count source: a labelsz index, or the instance it indexes.

neu_mark.dvid.open_points_source(src: Mapping[str, Any], *, prefer_locked: bool = False) → dict[str, Any]¶: Resolve and validate a point-annotation source. Call this before anything else.

neu_mark.dvid.open_source(spec: Mapping[str, Any], *, expect: str | tuple[str, ...], prefer_locked: bool = False) → dict[str, Any]¶

Resolve the ref, check the instance type, and return the pinned spec.

The returned spec carries the concrete uuid, never the ref, so a branch that moves mid-run cannot leave a table half from one node and half from the next.

neu_mark.dvid.refine(lo: str, hi: str) → list[str]¶

Boundaries subdividing [lo, hi] by appending one digit to lo.

'1'..``’2’`` becomes '1', '10', '11', … '19', '2', since decimal-string keys sort lexicographically. A range that times out is split this way rather than retried, because the proxy’s time limit is fixed while the server’s speed varies with load.

Candidates outside (lo, hi) are dropped, so boundaries stay sorted; [lo, hi] is returned unchanged when none falls strictly between them (it cannot be split further).

neu_mark.dvid.require_sync(source: Mapping[str, Any]) → list[str]¶

The labelmap(s) an annotation instance is synced to, or raise explaining why not.

/label/<body> relies on the sync’s index, so without one every per-body fetch would fail.

neu_mark.dvid.resolve_labelsz(source: Mapping[str, Any]) → dict[str, Any]¶

Point a source at the labelsz instance to query, from whatever was given.

Accepts the labelsz instance itself, or the annotation instance it indexes (e.g. synapses), found through the labelsz instance’s Base.Syncs.

neu_mark.dvid.BODY_INSTANCE = 'keyvalue'¶

neu_mark.dvid.COUNT_INSTANCE = 'labelsz'¶

neu_mark.dvid.DEFAULT_KEY_BOUNDARIES¶

neu_mark.dvid.DEFAULT_THREADS = 8¶

neu_mark.dvid.MAX_SPLIT_DEPTH = 2¶

neu_mark.dvid.ON_OVERLAP = ('warn', 'error')¶

neu_mark.dvid.POINT_INSTANCE = 'annotation'¶

neu_mark.dvid.ROI_UNSPECIFIED = '<unspecified>'¶

neu_mark.dvid.logger¶