neu_vol.ops.pack

Pack a small volume into one HDF5 file, described well enough to be written back.

The inverse of ops/write.py: that places a piece into a large volume, this produces the piece, recording where it belongs and at what scale so neu-vol write can place it without anyone re-typing an offset.

The source is anything readable: an image stack, an HDF5 file (whose recorded frame becomes the default), or a volume, in which case level and crop_start/crop_stop take a box out of one level and the crop origin becomes the recorded voxel_offset.

Recorded: voxel_offset (whole voxels) on the dataset; voxel_size / units / axes on the root and the dataset, so either alone is self-describing; offset (the same place in units) on the dataset. axes makes voxel_offset’s order knowable: the name is precomputed’s, where it means xyz, and a wrong guess mirrors the piece through the z=x diagonal.

Reads are blocked, so a volume larger than expected still packs. Serial and in-process.

Attributes

DEFAULT_BLOCK_BYTES
DEFAULT_DATASET
DEFAULT_OFFSET_FIELD
DEFAULT_VOXEL_SIZE_FIELD
FRAME_ATTRS
KIND_ATTR
SERVER_FORMATS
logger

Functions

check_fits(data, dtype[, region]) → None	Raise unless every value of data lies within dtype's range.
check_hdf5_target(out, name, frame, *[, overwrite, ...]) → tuple[dict, list[str], bool]	What an existing out allows: (its frame, its datasets, replacing).
hdf5_dataset_name(dataset) → str	A dataset name as a rooted path, defaulting to DEFAULT_DATASET.
pack_hdf5(src, out, *[, voxel_size, voxel_offset, ...]) → dict	Write src into out as an HDF5 dataset carrying its frame and position.
pack_hdf5_datasets(src, out[, datasets, src_format, ...]) → list[dict]	Repack several datasets of one HDF5 file into out, each under its own name.
resolve_source(src[, src_format, level, dataset, ...]) → tuple[dict, dict]	The spec for the level to read, and what the source says about its own frame.
write_hdf5_array(out, name, read_region, *, shape, ...) → None	Create out's name from read_region, and record the frame beside it.

Module Contents

neu_vol.ops.pack.check_fits(data: numpy.ndarray, dtype: Any, region: Sequence[slice] | None = None) → None

Raise unless every value of data lies within dtype’s range.

A cast that cannot lose range (uint16 -> uint64) is not inspected. Only range is checked: integer -> float precision and float -> integer fractions are not.

neu_vol.ops.pack.check_hdf5_target(out: str, name: str, frame: dict, *, overwrite: bool = False, voxel_size_field: str = DEFAULT_VOXEL_SIZE_FIELD) → tuple[dict, list[str], bool]

What an existing out allows: (its frame, its datasets, replacing).

An existing file is added to rather than replaced; each dataset keeps its own voxel_offset. Refused: a frame that disagrees with the file’s, and a dataset name already in use unless overwrite. A missing file answers with empties.

neu_vol.ops.pack.hdf5_dataset_name(dataset: str | None) → str

A dataset name as a rooted path, defaulting to DEFAULT_DATASET.

Rooted, the form h5py reports from visititems, so "gt/piece0" and "/gt/piece0" compare equal.

neu_vol.ops.pack.pack_hdf5(src: str | dict, out: str, *, voxel_size: Sequence[float] | None = None, voxel_offset: Sequence[int] | None = None, units: str | None = None, axes: Sequence[str] = ('z', 'y', 'x'), level: int = 0, crop_start: Sequence[int] | None = None, crop_stop: Sequence[int] | None = None, dataset: str | None = None, src_dataset: str | None = None, src_format: str | None = None, supervoxels: bool = False, prefer_locked: bool = False, wk_mapping: str | None = None, voxel_size_field: str = DEFAULT_VOXEL_SIZE_FIELD, offset_field: str = DEFAULT_OFFSET_FIELD, background: Sequence[int] | None = None, kind: str | None = None, dtype: str | None = None, truncate: bool = False, chunk: Sequence[int] | None = None, compression: str | None = 'gzip', overwrite: bool = False, dry_run: bool = False, max_bytes: int = DEFAULT_BLOCK_BYTES, warn_others: bool = True) → dict

Write src into out as an HDF5 dataset carrying its frame and position.

dataset defaults to DEFAULT_DATASET. An existing file is added to rather than replaced, provided its recorded frame matches; a dataset name already in use needs another dataset or overwrite=True.

voxel_offset is in the same axis order as axes, whole voxels, and names the position of the piece’s (0, ...) corner in the volume it belongs to.

kind records what the voxels mean (KIND_ATTR) and defaults to what the source recorded; an image stack or bare array records none, so pass it or leave it unrecorded. It is never guessed from the dtype.

A DVID or webknossos source (dvid:// / wk://) takes level like any pyramid, records its voxel size and kind, and takes the same read options as convert(): supervoxels and prefer_locked for DVID, wk_mapping for webknossos. Give it a crop; a whole instance is far too large for one file.

h5py reads and writes only filesystems, so an HDF5 src or an out on an object store (s3:// / gs://) is worked on as a temporary local copy: downloaded first (out only if it exists, so it can be added to), and out uploaded once the write succeeds. neu_vol.transfer says how the bytes move.

dtype narrower than the source’s is refused if any value falls outside its range (check_fits()); truncate=True casts anyway, as numpy’s astype does.

To write an array that is already in memory, use neu_vol.write_piece() — same file layout, no round trip through a store. To repack several datasets of one file, pack_hdf5_datasets().

neu_vol.ops.pack.pack_hdf5_datasets(src: str, out: str, datasets: Sequence[str] | None = None, *, src_format: str | None = None, dry_run: bool = False, **kwargs) → list[dict]

Repack several datasets of one HDF5 file into out, each under its own name.

datasets defaults to every volumetric dataset in src. Each keeps the frame and voxel_offset it records; the other pack_hdf5() arguments (dtype, truncate, chunk, compression, overwrite, …) apply to all of them. A remote src or out is transferred once for the whole set, not once per dataset.

neu_vol.ops.pack.resolve_source(src: str | dict, src_format: str | None = None, level: int = 0, dataset: str | None = None, voxel_size_field: str = 'voxel_size', offset_field: str = 'voxel_offset', options: Mapping[str, Any] | None = None) → tuple[dict, dict]

The spec for the level to read, and what the source says about its own frame.

options are the server-source read options of with_source_options(). Three kinds of source:

• a server source (DVID, webknossos) — its levels, voxel sizes and kind come from the server’s metadata, and the spec read is the one that metadata resolves: for a DVID branch ref, the concrete node, so every block comes from the same version.

• a multiscale volume — resolved through level_spec(), which addresses a level correctly in either format; its per-level voxel size, units and kind come back too.

• anything else (an image stack, a bare array, an HDF5 file): no levels, so a level other than 0 is an error. An HDF5 file’s recorded frame is read.

neu_vol.ops.pack.write_hdf5_array(out: str, name: str, read_region, *, shape: Sequence[int], dtype: Any, voxel_size: Sequence[float], voxel_offset: Sequence[int], units: str = 'nm', axes: Sequence[str] = ('z', 'y', 'x'), kind: str | None = None, chunk: Sequence[int] | None = None, compression: str | None = 'gzip', replacing: bool = False, block: Sequence[int] | None = None, has_channels: bool = False, voxel_size_field: str = DEFAULT_VOXEL_SIZE_FIELD, offset_field: str = DEFAULT_OFFSET_FIELD, truncate: bool = False) → None

Create out’s name from read_region, and record the frame beside it.

read_region(region) -> ndarray is called once per block, so the whole array is never held. voxel_offset describes the spatial axes; a leading channel axis gets 0. voxel_size / units / axes go on the root and the dataset; voxel_offset, offset and kind on the dataset alone (see the module docstring).

A value outside dtype’s range raises (check_fits()) unless truncate. The dataset is written under a temporary name and moved into place only once complete, so a refused or failed write leaves the file as it was, a replaced dataset included.

neu_vol.ops.pack.DEFAULT_BLOCK_BYTES = 1073741824

neu_vol.ops.pack.DEFAULT_DATASET = '/data'

neu_vol.ops.pack.DEFAULT_OFFSET_FIELD = 'voxel_offset'

neu_vol.ops.pack.DEFAULT_VOXEL_SIZE_FIELD = 'voxel_size'

neu_vol.ops.pack.FRAME_ATTRS = ('voxel_size', 'units', 'axes')

neu_vol.ops.pack.KIND_ATTR = 'kind'

neu_vol.ops.pack.SERVER_FORMATS = ('dvid', 'webknossos')

neu_vol.ops.pack.logger