neu_vol.retry¶
Bounded retry for transient object-store and DVID failures.
At the scale of a whole-volume copy, connections get reset, throttled or answered with 5xx even after TensorStore’s own request-level retries; a shared DVID server answers overload with 503.
Classification is by message text, since TensorStore raises both PERMISSION_DENIED
and UNAVAILABLE as ValueError. Permanent markers are checked first and win, so a
403 fails at once, and an unrecognized error is permanent, so a new failure mode
surfaces promptly.
Only wrap operations that are safe to repeat: writing a block to the same region is; appending to a manifest is not.
Attributes¶
Functions¶
|
Would retrying this error plausibly succeed? |
|
Call |
Module Contents¶
- neu_vol.retry.is_transient(exc: BaseException) bool¶
Would retrying this error plausibly succeed?
Permanent markers take precedence, and an unrecognized error is permanent.
- neu_vol.retry.with_retry(fn: Callable[[], T], *, attempts: int = 5, base_delay: float = 1.0, max_delay: float = 30.0, label: str = '', jitter: bool = True) T¶
Call
fn, retrying transient failures with exponential backoff.Raises the last exception once
attemptsis exhausted, and re-raises a permanent error immediately.jitterspreads retries across concurrent workers, so many workers hitting one throttle do not all return at once.
- neu_vol.retry.PERMANENT_MARKERS = ('PERMISSION_DENIED', 'NOT_FOUND', 'INVALID_ARGUMENT', 'FAILED_PRECONDITION', 'OUT_OF_RANGE',...¶
- neu_vol.retry.T¶
- neu_vol.retry.TRANSIENT_MARKERS = ('UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'CURL error', 'Connection...¶
- neu_vol.retry.logger¶