`neu-mark`¶

From neu-mark. DVID annotations into columnar tables, and on into neuroglancer: synapses and other point annotations, and the per-body records that carry a neuron’s name and status.

python -m neu_mark is equivalent.

Every fetch takes an explicit --bodies list, because DVID has no cheap way to enumerate the bodies worth asking about — select-bodies is how you build one, from the synapse-count index DVID already maintains.

Fetch DVID annotations into tables, and publish them to neuroglancer.

usage: neu-mark [-h] [--version] [-v] [--store-logs]
                {points,bodies,select-bodies,segment-properties,annotation-source,roi-layers,roi-survey,info,help}
                ...

Positional Arguments¶

cmd: Possible choices: points, bodies, select-bodies, segment-properties, annotation-source, roi-layers, roi-survey, info, help

Named Arguments¶

--version

show program’s version number and exit

-v, --verbose

debug logging

Default: False

--store-logs

do not filter the store’s own logging. The S3 driver emits two AuthCredentialsProvider lines per prefix at ERROR severity that are not failures — they record the providers that missed before the environment one succeeded — so they are filtered by default. PERMISSION_DENIED and AccessDenied are never filtered

Default: False

Sub-commands¶

points¶

Fetch every element of every listed body from a DVID annotation instance, and write two tables: points (one row per element) and relationships (one row per relationship, with the partner’s body resolved where it was fetched).

–all fetches the WHOLE instance by region instead, and looks up every element’s body in the synced labelmap at the same node, so every partner’s body is known. A few million elements fetch in under a minute. The body list still chooses what is meshed and named; annotation-source –listed marks which synapses touch it.

neu-mark points [-h] --src URL --out DST [--bodies IDS] [--body-column NAME]
                [--format {parquet,csv,feather}] [--dvid-locked] [--all]
                [--threads N] [--drop-unmatched] [--connections] [--rois LIST]
                [--strict-rois] [--roi-source URL]

Named Arguments¶

--src

DVID source as dvid://SERVER/UUID/INSTANCE, always three segments. SERVER may carry a port (emdata3:8900); a bare host gets http://, so use dvid+https://HOST/… for TLS. UUID may be a node uuid (93fdbc, abbreviated is fine) or a repo:branch ref (93fdbc:main), meaning HEAD of that branch. INSTANCE is the data instance name. A ‘:’ in the server or the uuid adds no segment — this splits on ‘/’ alone. Alternatively ‘@name’ looks the instance up in your config (neu_mark.config), and the resolved URL is printed so the command still says what it read. For this command INSTANCE is an annotation instance, e.g. ‘synapses’.

--out

where the tables go — a local directory or s3://bucket/prefix. May contain {uuid}, {uuid:N}, {uuid:full}, {branch} or {instance}, expanded from the resolved node.

--bodies

body ids: inline (123,456), or a path to a .csv/.parquet/.feather with a body column, or a text file with one id per line. Omit it only with –all.

--body-column

which column holds the ids, when –bodies is a table with several and none is named recognisably.

--format

Possible choices: parquet, csv, feather

table format (default: parquet, which is the only one that preserves uint64 body ids; csv warns about it).

Default: 'parquet'

--dvid-locked

read the newest LOCKED ancestor rather than what the ref points at now. A branch HEAD is normally an open, still-changing node, so a pull from it is not reproducible.

Default: False

--all

every element of the instance, with bodies from the labelmap, instead of a body list. Mutually exclusive with –bodies.

Default: False

--threads

concurrent requests (default 8; processes for –all’s region fetch). DVID is a shared service and answers overload with 503; raise this deliberately.

--drop-unmatched

drop relationships whose partner body was not resolved. Off by default: an unresolved partner and a dangling reference look identical, so read the reported match rate first.

Default: False

--connections

also write a connections table: the oriented, de-duplicated (tbar, psd) pairs derived from relationships.

Default: False

--rois

label each synapse with the neuropil it falls in, adding a roi column to the points table. A comma-separated list of DVID roi instance names, a path to a file with one per line, ‘@name’ for a set from your config, or SPEC.toml:LAYER for the ROIs of a roi-layers layer. There is deliberately NO default: the combined ROI volume is built by overwriting, so ‘all of them’ would attribute a point to whichever containing ROI came last.

--strict-rois

refuse if the chosen ROIs overlap. By default overlap is allowed and MEASURED: the later ROI wins, and the report says how many of your synapses sit in an intersection and are therefore attributed by ROI order alone. Use this if you need a strict partition — the dataset has subtracted variants (INP(-ATL)(L), PENP(-AMMC)) for that.

Default: False

--roi-source

the DVID node to read –rois from, as dvid://SERVER/UUID (default: –src’s node). For ROIs redrawn on a later node: the points and their body ids still come from –src.

bodies¶

Fetch the per-body records from a DVID keyvalue instance and write a bodies table, one row per body. Records are ragged, so the table is the union of the fields present.

–all reads the WHOLE instance, which is a different population and often the one you want: a >=10-synapse selection holds 117 glia while the instance holds 1,014, because most glia sit below any synapse threshold. It reads a partition of the key space rather than asking for the key list, which is unreliable at this size.

neu-mark bodies [-h] --src URL --out DST [--bodies IDS] [--body-column NAME]
                [--format {parquet,csv,feather}] [--dvid-locked] [--all]

Named Arguments¶

--src

--out

where the tables go — a local directory or s3://bucket/prefix. May contain {uuid}, {uuid:N}, {uuid:full}, {branch} or {instance}, expanded from the resolved node.

--bodies

body ids: inline (123,456), or a path to a .csv/.parquet/.feather with a body column, or a text file with one id per line. Omit it only with –all.

--body-column

which column holds the ids, when –bodies is a table with several and none is named recognisably.

--format

Possible choices: parquet, csv, feather

table format (default: parquet, which is the only one that preserves uint64 body ids; csv warns about it).

Default: 'parquet'

--dvid-locked

read the newest LOCKED ancestor rather than what the ref points at now. A branch HEAD is normally an open, still-changing node, so a pull from it is not reproducible.

Default: False

--all

every record in the instance, instead of a body list. Mutually exclusive with –bodies.

Default: False

select-bodies¶

Rank bodies by synapse count using DVID’s own labelsz index and write the list, for feeding to –bodies. Seconds, against ~70 minutes to scan every label’s voxel size — and synapse count is the better criterion anyway: a body with no synapses is a fragment, and many of the largest-by-volume bodies are glia.

neu-mark select-bodies [-h] --src URL --out FILE [--min-synapses N]
                       [--min-pre N] [--min-post N] [--limit N]
                       [--dvid-locked]

Named Arguments¶

--src

--out

the list’s path: .csv (read and edited by hand; every column is a non-nullable integer, which csv preserves), .parquet or .feather. A provenance record naming the node goes beside it as <name>.provenance.json. Accepts the same {uuid}/{branch}/{instance} placeholders.

--min-synapses

keep bodies with at least this many synapses in TOTAL (default 10). Total rather than per-type on purpose: sensory neurons may have no postsynapses, and a neuron leaving the traced volume may have no presynapses.

Default: 10

--min-pre

additionally require at least N presynapses. Off by default — it excludes neurons that only receive within the traced volume.

Default: 0

--min-post

additionally require at least N postsynapses. Off by default — it excludes sensory neurons.

Default: 0

--limit

keep only the top N after filtering. Results are ranked by total synapses, so this is how to ask for ‘the biggest N’.

--dvid-locked

rank from the newest LOCKED ancestor. Worth doing: body ids change with proofreading, so the list and the annotations should come from the same node.

Default: False

segment-properties¶

Build a segment_properties source from the per-body records and write it INTO a precomputed segmentation volume, so one neuroglancer layer shows each body’s name, its tags and its synapse counts.

The label is a field of the records, raw — the rules module’s LABEL, or –label — which is what lets this run before a cell-type parse is settled. Tags carry a facet prefix (type:, side:) because the format allows only ONE tags property, so every facet has to pool into it.

Facets come from the rules module –rules loads; nothing is built in, since what a name’s parts mean (side, column, glomerulus, cell type) depends on the dataset. Without –rules the document carries labels and numbers only. Run –dry-run first to see what each rule fired on.

neu-mark segment-properties [-h] --src URL --dst VOLUME
                            (--bodies IDS | --meshed-bodies)
                            [--body-column NAME] [--counts URL]
                            [--labelmap URL] [--synapses DIR]
                            [--roi-spec TOML] [--rules PATH] [--label FIELD]
                            [--drop RULE] [--max-tags N] [--table FILE]
                            [--dry-run] [--no-link] [--dvid-locked]

Named Arguments¶

--src

--dst

the precomputed SEGMENTATION VOLUME to write into (local or s3://…). The document lands at <dst>/segment_properties/info, and the volume’s own info gains the key pointing at it. Accepts the same {uuid}/{branch}/{instance} placeholders.

--bodies

body ids: inline, or a .csv/.parquet/.feather with a body column, or a text file with one per line.

--meshed-bodies

the bodies with a mesh in –dst, e.g. after neu-morpho run. Unsharded meshes only.

Default: False

--body-column

which column holds the ids, if the table has several.

--counts

a labelsz instance (or the annotation instance it indexes) to add pre, post and syn number properties. Cheap.

--labelmap

a labelmap instance to add a voxels number property, from DVID’s own /sizes. ~1 minute for 20k bodies.

--synapses

a directory an earlier points –rois run wrote. Gives the rules a synapse_neuropils column: per body, the share of its pre, post and all synapses in each ROI, so a rule can tag a cell by where it connects. Not itself published.

--roi-spec

a roi-layers spec, with –synapses: adds synapse_neuropils_top, the same shares grouped by the spec’s top-level neuropils.

--rules

a Python module of @rule functions, or @name for one configured under [rule_sets]: its RULES, or every @rule at module level in declaration order, which is also the order the viewer lists the facets in. Its LABEL names the label field.

--label

the record field to label segments with, overriding the module’s LABEL. With neither, segments are known by id.

--drop

exclude the bodies this rule fires on, rather than tagging them (repeatable), using the module’s own definition — e.g. –drop glia. An unknown rule name is refused.

--max-tags

cap the tag vocabulary. Below 1 this is a fraction of the segments written (the default is 0.10, floored at 100 tags so short lists are not judged by a ratio); 1 or above is an absolute count, respected as given. A rule echoing whole instance strings back produces a valid document no viewer can use; raise this when a dataset genuinely has that many distinct types.

--table

also write a row per requested body to FILE (.parquet, .csv or .feather; local or s3://…): its label, tags, one tag:<prefix> column per facet, one flag:<tag> per bare tag, the counts, and the record the rules read. Excluded bodies are kept, with the reason in excluded. Written with –dry-run too.

--dry-run

build and report without writing anything but –table. –no-link still writes the document; this writes nothing at all.

Default: False

--no-link

write the source WITHOUT adding the key to the volume’s info. Use this to inspect the output before touching a published volume; the linking step is what makes a viewer associate the two.

Default: True

--dvid-locked

read the newest LOCKED ancestor rather than the branch HEAD.

Default: False

annotation-source¶

Write a neuroglancer_annotations_v1 LINE source, one line from each presynaptic site to each of its partners. Lines, not points, because a connectome is edges — and one endpoint pair is one annotation, so a T-bar with five partners contributes five.

Input is the tables an earlier points run wrote (–tables), which is the cheap path: the tables are the durable artifact and a rebuild with different bounds or sharding need not refetch. Give –src and –bodies instead to fetch first.

This is a SEPARATE source from the segmentation — unlike mesh and skeletons, an annotation layer cannot be named from a volume’s info, so a viewer adds it as its own layer.

neu-mark annotation-source [-h] --dst PATH [--tables DIR] [--src URL]
                           [--bodies IDS] [--body-column NAME]
                           [--voxel-size X,Y,Z] [--bounds X0,Y0,Z0,X1,Y1,Z1]
                           [--drop-partial] [--listed IDS] [--roi-spec TOML]
                           [--per-cell N] [--threads N] [--dvid-locked]
                           [--verify [N]] [--dry-run]

Named Arguments¶

--dst

where the source goes (local or s3://…). Accepts the same {uuid}/{branch}/{instance} placeholders when –src is given.

--tables

a directory an earlier points run wrote, holding connections.parquet and points.parquet.

--src

DVID source as dvid://SERVER/UUID/INSTANCE, always three segments. SERVER may carry a port (emdata3:8900); a bare host gets http://, so use dvid+https://HOST/… for TLS. UUID may be a node uuid (93fdbc, abbreviated is fine) or a repo:branch ref (93fdbc:main), meaning HEAD of that branch. INSTANCE is the data instance name. A ‘:’ in the server or the uuid adds no segment — this splits on ‘/’ alone. Alternatively ‘@name’ looks the instance up in your config (neu_mark.config), and the resolved URL is printed so the command still says what it read. Only needed to fetch now instead of using –tables.

--bodies

body ids to fetch, when –src is given rather than –tables.

--body-column

which column holds the ids, if the table has several.

--voxel-size

nanometres per voxel of the frame the positions are in. Annotation coordinates are stored in VOXELS, and this is what puts them in the same physical space as the segmentation (default ‘8,8,8’).

Default: '8,8,8'

--bounds

the source’s lower and upper bound in voxels. Defaults to the data’s own extent, padded by one voxel.

--drop-partial

drop lines with only one endpoint’s body resolved. By default they are KEPT and undifferentiated: the partner is a real synapse whose body simply was not in the fetched set, and the line is still where the synapse is.

Default: True

--listed

the bodies the segmentation layer shows (meshed and named, e.g. the select-bodies list), in any –bodies form: adds pre_listed and post_listed, which the synapse shader’s hide_unlisted reads. Worth it for tables from points –all, where most partners are other bodies.

--roi-spec

a roi-layers spec: adds roi and neuropil properties, each synapse’s ROI and its top-level neuropil by the spec’s ids (the ids the ROI layers paint, so a synapse coloured by id matches its neuropil’s mesh). Needs tables from points –rois.

--per-cell

annotations a spatial cell aims to hold, which sets how many levels the grid needs and how much a zoomed-out view downloads (default 4000).

Default: 4000

--threads

concurrent DVID requests, when fetching rather than using –tables.

--dvid-locked

read the newest LOCKED ancestor rather than the branch HEAD.

Default: False

--verify

after writing, fetch N annotations back THROUGH the source’s own info and compare them with the table. Everything before the write can be checked in memory; only a read-back proves the keys are right, and a wrong key leaves a viewer with nothing while every byte on the store is correct.

--dry-run

report the plan — lines, stride, levels, shards — write nothing.

Default: False

roi-layers¶

Write a DVID node’s ROIs as precomputed segmentations, one per layer of a spec, each with segment properties naming its ROIs. A voxel holds one label, so ROIs that nest go in different layers, and overlaps between ROIs in one layer are settled by the spec’s overlap policy; ROIs nesting within one layer are refused.

The spec (TOML) lists the layers, a neuropil tree that the ‘top’ and ‘parts’ layers cut, ids shared by every layer, descriptions, and the ROIs to ignore. Every ROI on the node must be in a layer or ignored, so a new one is never dropped unseen. The layers are not meshed here: run neu-morpho run –stages index,mesh on each.

neu-mark roi-layers [-h] --src URL --spec TOML --like VOLUME --dst PATH
                    [--layers NAMES] [--overwrite] [--dvid-locked] [--dry-run]

Named Arguments¶

--src

the DVID node, as dvid://SERVER/UUID: ROIs are a node’s roi instances, so no instance is named. UUID may be a branch ref, resolved once and recorded.

--spec

the layer spec (see neu_mark.roi_layers).

--like

the segmentation the layers go beside. Its level-0 voxel size and origin set the grid: an ROI block is 32 of its voxels.

--dst

where the layers go, one volume per layer plus rois.json (local or s3://…).

--layers

comma-separated layer names to build, instead of all of them.

--overwrite

replace layers that already exist. Their meshes go too, so re-run the meshing.

Default: False

--dvid-locked

read the newest LOCKED ancestor rather than the branch HEAD.

Default: False

--dry-run

fetch and resolve every layer, report it, write nothing.

Default: False

roi-survey¶

Fetch every ROI on a DVID node and report what the blocks say: each ROI’s size and parent (the smallest larger ROI holding most of it), how well each parent’s children cover it and how far they stick out, the overlaps between ROIs that are not nested, empty and identical ROIs. The input to a roi-layers spec, and the check when a node’s ROIs change.

neu-mark roi-survey [-h] --src URL [--compare URL] [--spec TOML]
                    [--like VOLUME] [--contained FRAC] [--max-overlap FRAC]
                    [--out JSON] [--dvid-locked]

Named Arguments¶

--src

the DVID node, as dvid://SERVER/UUID. UUID may be a branch ref, resolved once and recorded.

--compare

another node, e.g. the one a spec was written against: report each ROI whose blocks changed, and any added or removed.

--spec

a roi-layers spec to check against this node: unplaced ROIs, parts outside their top-level neuropil, and each layer’s overlaps or why it would be refused.

--like

the segmentation the ROIs go beside: report ROIs reaching past its level-0 extent.

--contained

a child this far inside its parent is not reported as sticking out (default 0.95).

--max-overlap

flag siblings overlapping by more than this share of the smaller, as roi-layers refuses them (default 0.2).

--out

also write the full record, every overlapping pair included.

--dvid-locked

read the newest LOCKED ancestor rather than the branch HEAD.

Default: False

info¶

One request per fact: the instance type, what it is synced to, and both candidate nodes for the ref (what it points at now, and the newest locked one).

neu-mark info [-h] --src URL

Named Arguments¶

--src: DVID source as dvid://SERVER/UUID/INSTANCE, always three segments. SERVER may carry a port (emdata3:8900); a bare host gets http://, so use dvid+https://HOST/… for TLS. UUID may be a node uuid (93fdbc, abbreviated is fine) or a repo:branch ref (93fdbc:main), meaning HEAD of that branch. INSTANCE is the data instance name. A ‘:’ in the server or the uuid adds no segment — this splits on ‘/’ alone. Alternatively ‘@name’ looks the instance up in your config (neu_mark.config), and the resolved URL is printed so the command still says what it read.

help¶

Print the help for the tool, or for one of its subcommands.

neu-mark help explore and neu-mark explore –help are the same thing.

neu-mark help [-h] [COMMAND ...]

Positional Arguments¶

COMMAND: the subcommand to describe (default: the whole tool)

neu-mark¶

Positional Arguments¶

Named Arguments¶

Sub-commands¶

points¶

Named Arguments¶

bodies¶

Named Arguments¶

select-bodies¶

Named Arguments¶

segment-properties¶

Named Arguments¶

annotation-source¶

Named Arguments¶

roi-layers¶

Named Arguments¶

roi-survey¶

Named Arguments¶

info¶

Named Arguments¶

help¶

Positional Arguments¶

`neu-mark`¶