neu_vol.sharded¶

Writing and reading neuroglancer_uint64_sharded_v1 indexes.

A sharded index stores many keys in a few .shard files instead of one object each, which matters at scale: an annotation source with hundreds of millions of annotations, or a volume under an inode quota.

Writing goes entirely through tensorstore’s neuroglancer_uint64_sharded kvstore driver (write_all(), read_one()), so the format is not reimplemented for it. ShardReader is the one exception: the sharded multi-resolution mesh format stores fragment data outside the index, at byte offsets relative to the manifest, and the driver cannot say where an entry lies. So the addressing (hash, shard index, minishard table) is reimplemented here, for reading, and a test asserts it agrees with the driver.

Two things a caller has to get right:

Keys are 8-byte big-endian uint64 (key()); everything else in precomputed is little-endian.
The ``metadata`` block is the sharding spec that also goes into your ``info``. Pass the same dict to both.

shard_bits=0 with minishard_bits=0 writes a single 0.shard, which suits a small index.

Attributes¶

`AT_TYPE`
`DEFAULT_HASH`

Classes¶

ShardReader

Locates and reads entries of a sharded index, by key and by byte offset.

Functions¶

`clear`(location, *parts) → int	Delete the `.shard` objects of an index. Returns how many were removed.
`compressed_morton_code`(positions, grid_shape)	Compressed Morton codes for grid cell positions, as precomputed keys them.
`key`(chunk_id) → bytes	A uint64 key as the sharded kvstore expects it: 8 bytes, big-endian.
`morton_bits`(grid_shape) → list[int]	How many bits the compressed Morton code spends on each dimension.
`murmurhash3_x86_128_low64`(data[, seed]) → int	The low 64 bits of MurmurHash3 x86_128, the hash the sharded format specifies.
`open_kvstore`(location, metadata, *parts)	A sharded kvstore rooted at `location`/`parts`, local or remote.
`plan_sharding`(n_keys, *[, per_shard, max_shard_bits]) → dict[str, Any]	Sharding parameters for `n_keys` entries, aiming at `per_shard` keys per file.
`read_one`(location, metadata, chunk_id, *parts) → bytes \| None	One entry by id, or `None` if absent. For tests and for spot-checking output.
`shard_location`(chunk_id, metadata) → tuple[str, int]	`(shard filename, minishard number)` for a key, per the sharding spec.
`sharding_spec`(*, shard_bits, minishard_bits[, ...]) → dict[str, Any]	The sharding metadata, for both the kvstore and the `info` that describes it.
`write_all`(location, metadata, entries, *parts[, replace]) → int	Write `(chunk_id, bytes)` pairs into a sharded index. Returns how many.

Module Contents¶

class neu_vol.sharded.ShardReader(location: str | Mapping[str, Any], metadata: Mapping[str, Any], *parts: str)¶

Locates and reads entries of a sharded index, by key and by byte offset.

For the sharded multi-resolution mesh format, whose fragment data sits before the manifest at an offset the tensorstore driver cannot report. read_one() is the call for anything that only wants a value.

The shard and minishard indexes are cached on the instance, so reuse one reader across a batch: each shard’s indexes are then read once rather than once per body.

locate(chunk_id: int) → tuple[str, int, int] | None¶

(shard, offset, size) of the stored entry, or None if the key is absent.

size is the encoded length on disk (compressed, with data_encoding='gzip'), which is what the mesh format’s offsets are relative to.

read(chunk_id: int) → bytes | None¶: One entry by id, decoded per data_encoding; None if absent.

read_range(shard: str, start: int, end: int) → bytes | None¶: Raw bytes [start, end) of one shard file, undecoded.

location¶

metadata¶

parts = ()¶

neu_vol.sharded.clear(location: str | Mapping[str, Any], *parts: str) → int¶

Delete the .shard objects of an index. Returns how many were removed.

Not atomic: an interrupted rewrite leaves fewer shards than either version had. Only .shard objects are removed, because a sharded mesh or skeleton shares its directory with the info that declares it.

neu_vol.sharded.compressed_morton_code(positions, grid_shape: Sequence[int])¶

Compressed Morton codes for grid cell positions, as precomputed keys them.

positions is (n, 3) or (3,) in the same axis order as grid_shape, which for precomputed is xyz with x varying fastest. Returns an int64 array (or one int).

Not a row-major flattening: the two agree only while at most one axis is subdivided, so a small-grid test cannot tell them apart. Used for hand-built sharded indexes such as an annotation spatial index; tensorstore keys volume chunks itself.

neu_vol.sharded.key(chunk_id: int) → bytes¶: A uint64 key as the sharded kvstore expects it: 8 bytes, big-endian.

neu_vol.sharded.morton_bits(grid_shape: Sequence[int]) → list[int]¶

How many bits the compressed Morton code spends on each dimension.

ceil(log2(size)) per the spec, so zero for a size-1 dimension.

neu_vol.sharded.murmurhash3_x86_128_low64(data: bytes, seed: int = 0) → int¶

The low 64 bits of MurmurHash3 x86_128, the hash the sharded format specifies.

Pure Python: it runs once per key looked up, so a compiled dependency is not worth it. x86_128, not x64_128, which gives a different digest and sends every read to the wrong shard.

neu_vol.sharded.open_kvstore(location: str | Mapping[str, Any], metadata: Mapping[str, Any], *parts: str)¶

A sharded kvstore rooted at location/parts, local or remote.

The trailing slash the driver needs is added here.

neu_vol.sharded.plan_sharding(n_keys: int, *, per_shard: int = 30000, max_shard_bits: int = 10) → dict[str, Any]¶

Sharding parameters for n_keys entries, aiming at per_shard keys per file.

minishard_bits then keeps each minishard index small, since a reader fetching one key decodes a whole minishard index. A heuristic, not a rule from the spec.

neu_vol.sharded.read_one(location: str | Mapping[str, Any], metadata: Mapping[str, Any], chunk_id: int, *parts: str) → bytes | None¶: One entry by id, or None if absent. For tests and for spot-checking output.

neu_vol.sharded.shard_location(chunk_id: int, metadata: Mapping[str, Any]) → tuple[str, int]¶

(shard filename, minishard number) for a key, per the sharding spec.

The filename is lowercase hex zero-padded to ceil(shard_bits / 4) digits plus .shard; shard_bits=0 therefore gives 0.shard, the single-shard case.

neu_vol.sharded.sharding_spec(*, shard_bits: int, minishard_bits: int, preshift_bits: int = 0, hash: str = DEFAULT_HASH, data_encoding: str = 'gzip', minishard_index_encoding: str = 'gzip') → dict[str, Any]¶

The sharding metadata, for both the kvstore and the info that describes it.

2**shard_bits is the number of .shard files. minishard_bits splits each shard internally, bounding how much index a reader fetches to find one key.

neu_vol.sharded.write_all(location: str | Mapping[str, Any], metadata: Mapping[str, Any], entries: Iterable[tuple[int, bytes]], *parts: str, replace: bool = True) → int¶

Write (chunk_id, bytes) pairs into a sharded index. Returns how many.

One transaction for the whole batch, so each shard is written once rather than once per key.

replace (the default) deletes the existing shards first. Without it, keys already present and not rewritten stay, so an index whose key space changed would hold both generations behind unchanged object names. Pass replace=False only to add to an index on purpose.

neu_vol.sharded.AT_TYPE = 'neuroglancer_uint64_sharded_v1'¶

neu_vol.sharded.DEFAULT_HASH = 'murmurhash3_x86_128'¶