neu_vol.ops.progress

How far a conversion has got, per level: from its run manifest or from storage.

The manifest is what the run recorded: a local append-only JSONL, cheap to read even for a remote volume (1.5 s for 772k records, against tens of seconds of S3 listing). Storage is what is there, counted with one listing per level. They diverge if objects were deleted underneath, or if a different run wrote the volume.

Attributes

DONE_STATUSES

Functions

level_progress(volume, *[, progress_path, storage]) → dict	Blocks done per level of volume, against the number each level has.
manifest_counts(path) → dict[int, dict] | None	Per-level progress from a run's manifest, or None if absent.
other_manifests(manifest_path) → list[str]	Manifests beside manifest_path, such as a rebuild's .regen-from-<n> one.
stored_chunks(volume, fmt, level, scale_key) → int	Count the chunk objects stored for one level.
task_total(lv, chunk_total, chunk, level) → tuple[int, str]	The denominator for a level's manifest count, and a note if it isn't the chunk grid.

Module Contents

neu_vol.ops.progress.level_progress(volume: str, *, progress_path: str | None = None, storage: bool = False) → dict

Blocks done per level of volume, against the number each level has.

Counted from the run manifest (progress_path, defaulting to where the run puts it) unless storage is set or no manifest exists, then from a storage listing. Returns {"manifest_path", "source": "manifest"|"storage", "levels": [row, ...], "notes": [...]}, each row {"level", "shape", "chunk", "done", "total", "written", "empty"}; chunk is None where the level’s chunking is unknown, and written / empty are None when counting from storage.

Raises FileNotFoundError if nothing at volume is a volume, and ValueError for a single-array format, which has no chunks to count.

neu_vol.ops.progress.manifest_counts(path: str) → dict[int, dict] | None

Per-level progress from a run’s manifest, or None if absent.

{level: {"counts": {status: n}, "total": int|None, "task_shape": [...]|None, "grid": (max index + 1 per axis)}}.

The unit is the task, which may cover many destination chunks; the run records each level’s task total before dispatch. grid, the largest block index seen per axis, stands in for manifests written without one.

Statuses are kept apart rather than summed: empty means a block read as entirely the fill value, which is normal for sparse data and an alarm when it is every block.

A torn final line is normal — the writer may be appending — so it is skipped.

neu_vol.ops.progress.other_manifests(manifest_path: str) → list[str]

Manifests beside manifest_path, such as a rebuild’s .regen-from-<n> one.

neu_vol.ops.progress.stored_chunks(volume: str, fmt: str, level: int, scale_key: str | None) → int

Count the chunk objects stored for one level.

zarr v3 keeps each level’s chunks (or shards) under <level>/c/; precomputed keys each scale’s chunks under its scale key beside one shared info.

neu_vol.ops.progress.task_total(lv: dict, chunk_total: int, chunk: tuple, level: int) → tuple[int, str]

The denominator for a level’s manifest count, and a note if it isn’t the chunk grid.

The manifest counts tasks and chunk_total is the destination chunk grid. They differ at level 0 when the source reads in units coarser than the destination chunk. In order of preference: the recorded total; the grid implied by the largest recorded block index (exact once the level finished, a lower bound before); the chunk grid.

neu_vol.ops.progress.DONE_STATUSES = ('written', 'empty', 'skipped')