neu-morpho¶

Segment morphology from segmentation volumes: multi-resolution Draco-encoded meshes (via vol2mesh), written in neuroglancer-precomputed format, and measurement of published meshes and skeletons.

Orchestration across segments: blockrun (dask local/SLURM + resumable manifest).
Segmentation-array I/O: neu-vol (read precomputed / zarr / … regions and crop views).
Meshes: vol2mesh (marching cubes → simplify → Draco → multi-resolution neuroglancer mesh).
Skeletons are made by neu-skel, a separate package: GPL-licensed, because its tracers build on kimimaro and dijkstra3d. neu-skel run adds skeletons to a work dir this package’s run has indexed, writing them into the same volume, metrics DB and run summary. Everything that reads or measures skeletons — neu_vol.readback, neu-morpho measure skeletons, path distances — is here and needs no GPL code.

Large segments — whose binary mask over a big bounding box would OOM (the failure mode this package is designed around) — are meshed chunked and stitched rather than materialized whole.

Environment¶

One conda environment covers this repo and the two below it, each installed editable. blockrun and neu-vol must be sibling directories.

conda activate neu-env
pip install --no-deps -e ../blockrun -e ../neu-vol -e .
python -m pytest -q

Python is pinned to 3.12: vol2mesh and dvidutils are only built for py312 on flyem-forge, and they are conda-only (no PyPI equivalent), so the conda environment — not pip — has to provide them. The neu-suite README describes how to build the shared environment.

Running it¶

Installing the package provides the neu-morpho command (equivalently python -m neu_morpho):

neu-morpho run ...                  # the pipeline
neu-morpho progress <work-dir>      # live per-stage counts
neu-morpho run-report <work-dir>    # self-contained HTML summary
neu-morpho help [COMMAND]           # what --help prints, as a subcommand

run drives the stages against a dask cluster — local or SLURM, chosen by --config. Every stage is idempotent, so re-running the same command resumes.

The stages¶

--stages takes a comma-separated subset, run in this order. The default is index,mesh. Skeletons are a separate command, neu-skel run, against the same work dir.

stage	what it does
`seg`	Copies the ROI’s labels out as a precomputed volume, so the meshes and skeletons can be viewed against the segmentation they came from. Off by default — it is pure I/O and usually the most expensive stage.
`index`	Scans the volume once and records every body’s bounding box and voxel count in a SQLite metrics DB.
`mesh`	Multi-resolution Draco meshes, one per body.

index is what makes a size filter possible. It records every body’s voxel count and bounding box in the metrics DB, and --min-voxels / --limit-bodies build the allowlist from those counts; without the DB there is no way to say “only bodies above N voxels”. mesh and neu-skel’s skel work block by block and read no bounding box, so with an explicit --allowlist (or none) they run without it. It writes no volume data.

Once the DB exists you can re-run the later stages without it: --stages mesh is the normal way to redo meshing after a parameter change. mesh runs in two phases — fragments per (body, block), then one output assembled per body — so a re-run of only stage 2 reuses the fragments already on disk instead of recomputing them.

# 0. look at the pyramid and pick scales from real metadata (no cluster, no writes)
neu-morpho run --src /path/to/seg --describe

# 1. one small cube, in-process — the fastest way to see it work end to end
neu-morpho run --src ... \
    --dst /path/to/morpho/segmentation --work-dir /path/to/morpho \
    --serial --roi 0,0,0,512,2048,2048 --roi-scale 2 --stages index,mesh

# 2. the same cube on SLURM, surviving logout, publishing to s3
nohup env PYTHONUNBUFFERED=1 neu-morpho run --src ... \
    --dst s3://bucket/sample3/segmentation --work-dir /path/to/morpho \
    --config dask-slurm-example --config ~/my-site.yaml --workers 48 \
    --roi 0,0,0,512,2048,2048 --roi-scale 2 --stages index,mesh > run.log 2>&1 &
squeue -u "$USER"

# 3. widen or drop --roi for the whole volume; step 2's work is reused

PYTHONUNBUFFERED=1 is the console-script equivalent of python -u; without it the log lags a long run in 8 KB blocks. Keep that log off a shared network filesystem such as CephFS: unbuffered, every line is its own write, and once another node reads the file (tail -f) each one becomes a synchronous round trip. A local disk or your home directory is fine.

--roi-scale is required whenever --roi is given: the same six numbers name a box four times smaller per pyramid level, and the wrong one silently processes the wrong region instead of failing.

The same run from Python, where plan checks the arguments against the source and counts blocks without writing anything:

from neu_morpho import pipeline

p = pipeline.plan(src, "/path/to/morpho/segmentation", work_dir="/path/to/morpho",
                  roi=(0, 0, 0, 512, 2048, 2048), roi_scale=2)
summary = pipeline.run(p)            # in-process; pass client= to run on a dask cluster

Cluster config¶

--config takes a bundled template name or a path, and is repeatable, deep-merged left to right. The templates (dask-local, used by default, and dask-slurm-example) ship with blockrun, next to start_dask, so every consumer shares one set. An overlay carries only the keys that differ:

neu-morpho run --config dask-slurm-example --config ~/my-site.yaml ...

# my-site.yaml — everything else comes from the template
jobqueue:
  slurm:
    account: my-account
    queue: my-partition
    log-directory: /path/with/room

Unrecognised keys raise rather than merging silently, and the effective merged config is written into the run’s work dir. Site-specific configs and dataset allowlists are deliberately not in this repo.

Two destinations: `--dst` and `--work-dir`¶

--dst is the published volume — the precomputed labels with meshes and skeletons written inside it, so a single neuroglancer layer carries all three. It may be a local path or an s3:// URL.

precomputed://file:///path/to/morpho/segmentation
precomputed://s3://bucket/sample3/segmentation

That volume’s info gets "mesh": "mesh" and "skeletons": "skeleton" (the precomputed spec’s subdirectory-naming keys), and the copy carries voxel_offset so it lands at its true global position rather than at the origin.

--work-dir is everything else — metrics.db, manifests, failure logs and the stage-1 fragment stores. It must be an ordinary filesystem path: it holds a sqlite DB, appended JSONL and a fragment store read back with file I/O, none of which work over an object store. It defaults to the parent of a local --dst, and is required when --dst is remote.

--dst changed meaning. It used to be the run root, with the volume at <dst>/segmentation. It is now the volume itself. An existing invocation passing --dst /path/to/morpho will publish to .../morpho directly rather than .../morpho/segmentation. Add /segmentation to keep the old layout — the work dir then defaults to the old run root.

Because bookkeeping no longer lives inside the output, a manifest can outlive the data it describes: clear --dst and a resumed run would skip every task and report success having written nothing. Each stage refuses to resume when its manifest records completed work but its output info is gone, naming --no-resume as the explicit override.

A resume also keeps to the bodies stage 1 was run for. A completed block is not revisited, so a body added to the allowlist afterwards would get nothing; the run refuses instead. Mesh or skeletonize new bodies with a new --work-dir. Blocks are recorded by index, so a resume also refuses a different --block or scale, whose block (0, 0, 0) is a different region, and different --smoothing, whose finished blocks were meshed without it; change any of them in a new --work-dir, or with --no-resume.

--dry-run reports the plan (scales, voxel sizes, block counts per stage) without touching anything — worth running before any large job.

The ROI filters blocks on the global grid, it does not re-tile. A block’s index, its region and its nm coordinates are the same numbers in a small run and in the full run, so widening the ROI later reuses the fragments and manifest entries already on disk instead of redoing them. The caveat: a body straddling the ROI edge is built only from the blocks inside it, so it stays truncated until its neighbours are run.

Scales are integers; voxel sizes are read from the source. --mesh-scale 2 looks up that level’s actual resolution in the precomputed info (or the zarr OME metadata) rather than assuming 2**scale. Real pyramids are often anisotropic — z frequently isn’t downsampled at all — and that assumption is exactly what misaligns skeletons against meshes.

Size the cluster to one block, not one body. That is the whole point of the block-first design: peak memory is a single block’s mask and its distance transform regardless of how large the bodies in it are, so a constraint pinning jobs to fat nodes buys nothing and only shrinks the pool of nodes the job is eligible for. Read the sizing comments in blockrun’s dask-slurm-example template before changing --block.

When something fails¶

Task granularity differs per stage, and so does what a failure costs — the asymmetry is deliberate:

stage	one task is	on failure
`chunk` / `skel-chunk`	one block	raise, abort the stage
`assemble` / `skel-fuse`	one body	record `failed`, continue

Stage 2 isolates, because bodies are independent: a bad one costs exactly that body, is recorded in the manifest, is logged with its traceback to <work-dir>/failures.{mesh,skel}.jsonl, and is retried on the next run. One pathological body cannot kill hour 11 of a 50k-body run.

Stage 1 does not, and that is the point. Stage 2 aggregates across blocks, so a silently skipped block does not leave a hole in one block — it truncates every body passing through it and erases outright any body lying wholly inside it, while the output still looks complete. Stage 2 cannot distinguish “this body had 3 fragments” from “it had 4 and one block died”. A crash you notice; a truncated neuron you may not.

Isolation is right for one odd body and wrong for a broken environment — 40k identical failures are one data point, not 40k. So a stage also aborts on a systemic exception (MemoryError, ImportError, and OSError with ENOSPC / EDQUOT / EROFS, all of which recur for every remaining task by construction) or after max_consecutive_failures (default 10, one success resets, 0 disables). Ctrl-C always stops a run immediately. The CLI logs failed body ids per stage and exits non-zero, so a scripted pipeline cannot mistake a partial result for a clean one. --allow-body-failures makes it exit 0 when per-body failures are all that went wrong, for a pipeline that should carry on and retry them later; an aborted stage still exits non-zero.

One caveat on the metrics¶

mesh_area_nm2 is measured on the decimated mesh, since stage 1 simplifies before writing fragments (decimation_fraction defaults to 0.1). It is a sound relative size measure across bodies; do not read it as an absolute membrane area without checking the bias against decimation_fraction=1.0 on a sample.

License¶

Licensed under the Apache License, Version 2.0. See CONTRIBUTING.md for how contributions are licensed.