neu_vol.location

Destination/source locations: local paths, s3:///gs:// URLs, kvstore dicts.

Everything downstream speaks TensorStore kvstore specs (plain dicts), so the ops behave identically for local files and object stores. This module normalizes a user-facing location into a kvstore and joins subpaths uniformly.

Functions

as_directory(kvstore) → dict[str, Any]	A copy of kvstore whose path ends in /, for a driver given it as a root.
default_progress_path(dst) → str	Default manifest path for a destination (local: alongside it; remote: cwd).
ensure_credentials(kvstore) → dict[str, Any]	Bootstrap object-store credentials for a kvstore that is about to be opened.
exists(location, *parts) → bool	True if the object exists. One request; safe against object stores.
is_local(location) → bool	True if location resolves to an ordinary filesystem path, for callers that need
join(kvstore, *parts) → dict[str, Any]	Append path segments to a kvstore's path (leading '/' kept for files).
list_keys(location, *parts[, limit]) → list[str]	Key names directly under a prefix.
local_path(location) → str	The filesystem path for a local location; raises if it is remote.
read_bytes(location, *parts) → bytes | None	Read one object; None if it does not exist.
read_json(location, *parts)	Read and parse a JSON object; None if it does not exist.
read_range(location, start, end, *parts) → bytes | None	Read the half-open byte range [start, end) of one object; None if absent.
spec_kvstore(spec) → dict[str, Any]	kvstore for a backend spec (handles 'kvstore', 'path', and scheme URLs).
to_kvstore(location) → dict[str, Any]	Normalize a location to a TensorStore kvstore spec.
write_bytes(location, data, *parts) → None	Write one object, creating parent directories for the file driver.
write_json(location, obj, *parts[, indent]) → None	Serialize obj as JSON and write it.

Module Contents

neu_vol.location.as_directory(kvstore: Mapping[str, Any]) → dict[str, Any]

A copy of kvstore whose path ends in /, for a driver given it as a root.

neu_vol.location.default_progress_path(dst: str | Mapping[str, Any]) → str

Default manifest path for a destination (local: alongside it; remote: cwd).

A manifest is appended to and fsynced, so for a remote destination it lands in the working directory. Its name then carries a digest of the whole location beside the last component (seg.4f2a91c7.progress.jsonl), so two destinations ending in the same word never share a manifest, which would make each resume skip the other’s blocks. The digest covers driver, bucket and path only, trailing slash removed, so one destination always maps to one manifest.

neu_vol.location.ensure_credentials(kvstore: Mapping[str, Any]) → dict[str, Any]

Bootstrap object-store credentials for a kvstore that is about to be opened.

Call it wherever a store is opened, not where a spec is built: tensorstore’s S3 profile provider cannot read ~/.aws/credentials, so the profile is copied into AWS_* environment variables, per process. A dask worker that skipped it would get 403s while others succeed.

neu_vol.location.exists(location: str | Mapping[str, Any], *parts: str) → bool

True if the object exists. One request; safe against object stores.

neu_vol.location.is_local(location: str | Mapping[str, Any]) → bool

True if location resolves to an ordinary filesystem path, for callers that need POSIX semantics (sqlite, appended logs) to refuse an object store up front.

neu_vol.location.join(kvstore: Mapping[str, Any], *parts: str) → dict[str, Any]

Append path segments to a kvstore’s path (leading ‘/’ kept for files).

neu_vol.location.list_keys(location: str | Mapping[str, Any], *parts: str, limit: int | None = None) → list[str]

Key names directly under a prefix.

limit truncates the result, not the request: tensorstore enumerates the whole prefix first, so a prefix with millions of objects costs the same either way. To ask about one object, probe it by name with exists(); to bound a listing, list a narrower prefix.

neu_vol.location.local_path(location: str | Mapping[str, Any]) → str

The filesystem path for a local location; raises if it is remote.

neu_vol.location.read_bytes(location: str | Mapping[str, Any], *parts: str) → bytes | None

Read one object; None if it does not exist.

neu_vol.location.read_json(location: str | Mapping[str, Any], *parts: str)

Read and parse a JSON object; None if it does not exist.

neu_vol.location.read_range(location: str | Mapping[str, Any], start: int, end: int, *parts: str) → bytes | None

Read the half-open byte range [start, end) of one object; None if absent.

A range reaching past the end of the object raises OUT_OF_RANGE (a ValueError) on both the file and S3 drivers. On an object store this is an HTTP Range request.

neu_vol.location.spec_kvstore(spec: Mapping[str, Any]) → dict[str, Any]

kvstore for a backend spec (handles ‘kvstore’, ‘path’, and scheme URLs).

neu_vol.location.to_kvstore(location: str | Mapping[str, Any]) → dict[str, Any]

Normalize a location to a TensorStore kvstore spec.

Accepts a local path (made absolute, ~ expanded), an s3://bucket/prefix / gs://bucket/prefix URL, or an existing kvstore dict.

neu_vol.location.write_bytes(location: str | Mapping[str, Any], data: bytes, *parts: str) → None

Write one object, creating parent directories for the file driver.

neu_vol.location.write_json(location: str | Mapping[str, Any], obj: Any, *parts: str, indent: int | None = 2) → None

Serialize obj as JSON and write it.