neu_vol.retry

Bounded retry for transient object-store and DVID failures.

At the scale of a whole-volume copy, connections get reset, throttled or answered with 5xx even after TensorStore’s own request-level retries; a shared DVID server answers overload with 503.

Classification is by message text, since TensorStore raises both PERMISSION_DENIED and UNAVAILABLE as ValueError. Permanent markers are checked first and win, so a 403 fails at once, and an unrecognized error is permanent, so a new failure mode surfaces promptly.

Only wrap operations that are safe to repeat: writing a block to the same region is; appending to a manifest is not.

Attributes

Functions

is_transient(exc) → bool

Would retrying this error plausibly succeed?

with_retry(fn, *[, attempts, base_delay, max_delay, ...]) → T

Call fn, retrying transient failures with exponential backoff.

Module Contents

neu_vol.retry.is_transient(exc: BaseException) → bool

Would retrying this error plausibly succeed?

Permanent markers take precedence, and an unrecognized error is permanent.

neu_vol.retry.with_retry(fn: Callable[[], T], *, attempts: int = 5, base_delay: float = 1.0, max_delay: float = 30.0, label: str = '', jitter: bool = True) → T

Call fn, retrying transient failures with exponential backoff.

Raises the last exception once attempts is exhausted, and re-raises a permanent error immediately. jitter spreads retries across concurrent workers, so many workers hitting one throttle do not all return at once.

neu_vol.retry.PERMANENT_MARKERS = ('PERMISSION_DENIED', 'NOT_FOUND', 'INVALID_ARGUMENT', 'FAILED_PRECONDITION', 'OUT_OF_RANGE',...
neu_vol.retry.T
neu_vol.retry.TRANSIENT_MARKERS = ('UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'CURL error', 'Connection...
neu_vol.retry.logger