Skip to content

Manifest

timenet.manifest source

The dataset manifest: a frozen :class:Manifest and its JSON codec.

FilePart dataclass source

One data file of a dataset version. It has a path, a checksum, and a byte size in one record.

The path, checksum, and size stay together in one record. A reader does not need to join a file to its digest across two structures. A consumer can verify integrity and plan a download from the manifest alone.

checksum instance-attribute source

checksum: str

The digest of the file, with a sha256: prefix.

path instance-attribute source

path: str

The version-relative POSIX path to the file.

size instance-attribute source

size: int

The size of the file, in bytes.

Manifest dataclass source

The single source of truth a consumer reads to interpret a dataset version.

Raises:

Type Description
TimeNetInvalidManifestError

If timef_format_version is not a supported version.

SUPPORTED_FORMAT_VERSIONS class-attribute source

SUPPORTED_FORMAT_VERSIONS: frozenset[int] = frozenset({1})

build_env class-attribute instance-attribute source

build_env: dict[str, Any] = field(default_factory=dict)

The Python version and package set that produced this version.

Provenance only: nothing reads it to interpret the data. It is here so a builder can answer what produced a dataset version without re-deriving it from a build log.

counts class-attribute instance-attribute source

counts: ManifestCounts = field(
    default_factory=ManifestCounts
)

Row and entity counts recorded for quick inspection.

dataset_id instance-attribute source

dataset_id: str

A denormalized copy of metadata.dataset_id. A reader can get the id without parsing metadata.

files instance-attribute source

files: ManifestFiles

Descriptor (path, checksum, size) for every data artifact, grouped by kind.

metadata instance-attribute source

metadata: DatasetMetadata

Descriptive identity of the dataset (name, version, license, domains, tags).

schema class-attribute instance-attribute source

schema: DatasetSchema = field(default_factory=DatasetSchema)

Structural schema: time-series specs, annotations, and tasks.

timef_format_version class-attribute instance-attribute source

timef_format_version: int = 1

The TimeF manifest format version. The value must be in SUPPORTED_FORMAT_VERSIONS.

value_encoding class-attribute instance-attribute source

value_encoding: dict[str, str] = field(default_factory=dict)

spec_type -> the values-column encoding that its shards carry.

This field exists only for provenance. Parquet already records the applied encoding in each file's footer, so a reader does not need this field. The field lets a builder see what a build chose without opening a shard. The field is empty for a backend with no such choice.

values_backend class-attribute instance-attribute source

values_backend: str = ValuesBackend.PARQUET

Storage backend for the time-series values plane.

from_dict classmethod source

from_dict(data: dict[str, Any]) -> Manifest

Parse a manifest dict. Allow missing optional blocks.

Parameters:

Name Type Description Default
data dict[str, Any]

The manifest dict, for example the output of json.loads.

 required

Returns:

Type Description
Manifest

The parsed :class:Manifest.

Raises:

Type Description
TimeNetInvalidManifestError

If a required key is missing or a block is invalid.

from_json classmethod source

from_json(text: str) -> Manifest

Parse a manifest from a JSON string.

Parameters:

Name Type Description Default
text str

The JSON text.

 required

Returns:

Type Description
Manifest

The parsed :class:Manifest.

Raises:

Type Description
TimeNetInvalidManifestError

If the text is not valid JSON or a block is invalid.

to_dict source

to_dict() -> dict[str, Any]

Serialize the manifest to a JSON-compatible dict with all keys present.

Returns:

Type Description
dict[str, Any]

The canonical dict form.

to_json source

to_json() -> str

Serialize the manifest to a pretty JSON string.

Returns:

Type Description
str

The JSON text.

ManifestCounts dataclass source

Row/entity counts recorded in the manifest for quick inspection without opening the parquet.

annotations class-attribute instance-attribute source

annotations: int = 0

Number of unique annotation ids across all records.

records class-attribute instance-attribute source

records: int = 0

Total number of records in the dataset.

registered_annotations class-attribute instance-attribute source

registered_annotations: int = 0

Number of task-referenced annotations that no record carries. Lets the reader skip the annotation scan that recovers them when there are none.

tasks class-attribute instance-attribute source

tasks: dict[str, int] = field(default_factory=dict)

Count of tasks keyed by task type.

time_series_chunks class-attribute instance-attribute source

time_series_chunks: int = 0

Number of time series chunk placements written to parquet.

time_series_index_rows class-attribute instance-attribute source

time_series_index_rows: int = 0

Number of rows in the time series index.

time_series_specs class-attribute instance-attribute source

time_series_specs: dict[str, int] = field(
    default_factory=dict
)

Count of unique time series keyed by spec type.

ManifestFiles dataclass source

Descriptors for every artifact of a dataset version, grouped by kind. Readers use this data, not a glob.

Each artifact is a list of parts. This lets any artifact shard later without a change to the manifest format. Today the writer creates one part for records, annotations, and time_series_index. tasks and time_series already have several parts. Each part is a :class:FilePart object, with its own path, checksum, and size.

annotations instance-attribute source

annotations: tuple[FilePart, ...]

Parts of the annotations table.

records instance-attribute source

records: tuple[FilePart, ...]

Parts of the records table.

tasks class-attribute instance-attribute source

tasks: tuple[FilePart, ...] = ()

Parts of the task tables. There is one table for each task type.

time_series class-attribute instance-attribute source

time_series: tuple[FilePart, ...] = ()

Parts of the time series data. Each part is also a shard.

time_series_index instance-attribute source

time_series_index: tuple[FilePart, ...]

Parts of the time series index table.

all_files source

all_files() -> tuple[FilePart, ...]

Return every file descriptor across all artifacts, in a stable order.

Returns:

Type Description
FilePart

The parts of records, annotations, time_series_index, tasks, and
...

time_series, joined into one tuple.

all_parts source

all_parts() -> tuple[str, ...]

Return the version-relative path of every file, in the same order as :meth:all_files.

Returns:

Type Description
str

The path of every file. Use this when you only need to find the files, for example to
...

download them.