neu_vol.transfer

Whole-file transfers between a local path and an object store (s3:// / gs://).

For formats with no object-store driver (HDF5): download, work on the local copy, upload. S3 goes through boto3 when it is installed (the s3 extra): streamed to and from disk, multipart and parallel. Otherwise, and for GCS, through tensorstore, which holds the whole object in memory and uploads it in a single request — and S3 caps a single PUT at 5 GB.

Attributes

S3_SINGLE_PUT_BYTES

Functions

download(url, path) → None	Copy the object at url to the local file path. Raises if it does not exist.
upload(path, url) → None	Copy the local file path to url, replacing any object there.

Module Contents

neu_vol.transfer.download(url: str, path: str) → None

Copy the object at url to the local file path. Raises if it does not exist.

neu_vol.transfer.upload(path: str, url: str) → None

Copy the local file path to url, replacing any object there.

neu_vol.transfer.S3_SINGLE_PUT_BYTES = 5000000000