neu_mark.io

Writing tables out, and the provenance sidecar that makes them meaningful.

Parquet is the default because csv loses the type of a nullable integer column such as relationships.to_body (null where the partner was not fetched): pandas reads it back as str or float64, and float64 rounds ids above 2^53, so joins on it quietly find nothing. Parquet also carries the provenance record in its own metadata. csv remains for reading in a terminal and warns about such columns; feather is offered for fast local scratch.

Every write goes through neu_vol.location, so a destination may be a local path or s3://….

Attributes

FORMATS
METADATA_KEY
READ_FORMATS
logger

Functions

read_embedded_provenance(src, name) → dict | None	The provenance record stored inside a parquet file, if it is there.
read_table(src, name, *[, fmt])	Read a table back, for tests and for the stage that consumes these files.
read_tables(src, names) → tuple	Read several tables out of one directory, detecting each one's format.
split_table_path(path) → tuple[str, str, str]	(directory, name, fmt) of a table file path, the format from its extension.
table_name(name, fmt) → str
write_provenance(dst, record, *[, name]) → None	Write <name>.json beside the tables (csv and feather cannot embed it). A failure
write_table(df, dst, name, *[, fmt, metadata]) → str	Serialise df and write it under dst. Returns the key written.

Module Contents

neu_mark.io.read_embedded_provenance(src: str, name: str) → dict | None

The provenance record stored inside a parquet file, if it is there.

neu_mark.io.read_table(src: str, name: str, *, fmt: str = 'parquet')

Read a table back, for tests and for the stage that consumes these files.

neu_mark.io.read_tables(src: str, names: Sequence[str]) → tuple

Read several tables out of one directory, detecting each one’s format.

neu_mark.io.split_table_path(path: str) → tuple[str, str, str]

(directory, name, fmt) of a table file path, the format from its extension.

"out/bodies.csv" -> ("out", "bodies", "csv"). Works for s3:// URLs too.

neu_mark.io.table_name(name: str, fmt: str) → str

neu_mark.io.write_provenance(dst: str, record: Mapping[str, Any], *, name: str = 'provenance') → None

Write <name>.json beside the tables (csv and feather cannot embed it). A failure is logged, not raised. The record shape is neu-vol’s ops/provenance.py.

neu_mark.io.write_table(df, dst: str, name: str, *, fmt: str = 'parquet', metadata: Mapping[str, Any] | None = None) → str

Serialise df and write it under dst. Returns the key written.

neu_mark.io.FORMATS = ('parquet', 'csv', 'feather')

neu_mark.io.METADATA_KEY = b'neu_mark_provenance'

neu_mark.io.READ_FORMATS = ('parquet', 'feather', 'csv')

neu_mark.io.logger