Skip to content

Format

timenet.format source

The TimeF on-disk format contract: filenames, layout templates, and pinned Arrow schemas.

The writer and the reader both use this module. They are otherwise independent. The definitions that both sides of the round-trip must agree on live here, so neither side imports the other.

checksums source

File checksums recorded in the manifest and verified on read.

The reader must hash files the same way the writer did, or :meth:~timenet.reader.TimeFReader.verify means nothing. The algorithm and the block size live here so both sides call the same code.

CHECKSUM_PREFIX module-attribute source

CHECKSUM_PREFIX = 'sha256:'

file_checksum source

file_checksum(path: Path) -> str

Return a file's manifest checksum, hashing it a block at a time.

Parameters:

Name Type Description Default
path Path

The file to hash.

 required

Returns:

Type Description
str

The checksum as "sha256:<hex>".

stream_checksum source

stream_checksum(handle: _Readable) -> str

Return an open binary stream's manifest checksum, hashing it a block at a time.

This is the stream counterpart of :func:file_checksum. A reader can use it to hash a file opened through a pyarrow filesystem (local now, an object store later) instead of a local Path.

Parameters:

Name Type Description Default
handle _Readable

An open binary stream positioned at the start.

 required

Returns:

Type Description
str

The checksum as "sha256:<hex>".

constants source

On-disk filenames and layout templates for the TimeF format.

ANNOTATIONS_SORT_KEY module-attribute source

ANNOTATIONS_SORT_KEY = 'id'

ANNOTATIONS_TEMPLATE module-attribute source

ANNOTATIONS_TEMPLATE = 'annotations/part-{:08d}.parquet'

DEFAULT_CHUNK_MAX_BYTES module-attribute source

DEFAULT_CHUNK_MAX_BYTES = 1 * 2 ** 20

DEFAULT_COMPRESSION module-attribute source

DEFAULT_COMPRESSION = 'zstd'

DEFAULT_CONTROL_SHARD_TARGET_BYTES module-attribute source

DEFAULT_CONTROL_SHARD_TARGET_BYTES = 128 * 2 ** 20

DEFAULT_ROW_GROUP_TARGET_BYTES module-attribute source

DEFAULT_ROW_GROUP_TARGET_BYTES = 4 * 2 ** 20

DEFAULT_SHARD_TARGET_BYTES module-attribute source

DEFAULT_SHARD_TARGET_BYTES = 128 * 2 ** 20

INDEX_SORT_KEY module-attribute source

INDEX_SORT_KEY = 'record_id'

INDEX_TEMPLATE module-attribute source

INDEX_TEMPLATE = 'time_series_index/part-{:08d}.parquet'

MANIFEST_FILE module-attribute source

MANIFEST_FILE = 'manifest.json'

MAX_PART_INDEX module-attribute source

MAX_PART_INDEX = 10 ** PART_INDEX_DIGITS - 1

PART_INDEX_DIGITS module-attribute source

PART_INDEX_DIGITS = 8

RECORDS_TEMPLATE module-attribute source

RECORDS_TEMPLATE = 'records/part-{:08d}.parquet'

SHARD_DIR module-attribute source

SHARD_DIR = 'time_series'

SHARD_TEMPLATE module-attribute source

SHARD_TEMPLATE = 'time_series/part-{:08d}.parquet'

TASKS_DIR module-attribute source

TASKS_DIR = 'tasks'

TASK_PART_TEMPLATE module-attribute source

TASK_PART_TEMPLATE = (
    "tasks/task={task_type}/part-{:08d}.parquet"
)

check_relative_path source

check_relative_path(name: str, path: str) -> None

Reject a path that would resolve outside the dataset root it is meant to stay under.

Callers join a manifest file entry and a registry handle's relpath onto a root directory (or filesystem prefix) as-is. An absolute path or a .. segment would escape that root instead of raising. This would let a crafted manifest or caller read or write outside the intended directory.

Parameters:

Name Type Description Default
name str

The name of the field to check. It appears in the error message.

 required
path str

The path to check. It must be relative and rooted inside the dataset.

 required

Raises:

Type Description
TimeFValidationError

If path is absolute, or has a .. segment.

part_path source

part_path(template: str, index: int, **fields: str) -> str

Render a numbered part/shard path, refusing an index that would overflow the fixed width.

The reader visits parts in the order the manifest lists them. The zero-padded names keep an ls or a prefix listing in that same order. That order only holds while every number fits in PART_INDEX_DIGITS. A ninth digit sorts before the eighth and silently breaks it. Routing every part name through here turns that overflow into a loud error instead.

Parameters:

Name Type Description Default
template str

A layout template whose part number is the {:08d} field, for example SHARD_TEMPLATE.

 required
index int

The zero-based part number to render.

 required
fields str

Any remaining named fields the template needs, for example task_type.

 {}

Returns:

Type Description
str

The formatted version-relative path.

Raises:

Type Description
TimeFValidationError

If index is negative or needs more than PART_INDEX_DIGITS digits.

schemas source

Arrow schemas for the TimeF on-disk files, parameterized by id storage type.

The column layout is fixed. Only the id columns vary. Every id column holds one of the six logical ids in :data:LOGICAL_IDS. Each is stored as pa.string(), or as pa.binary(16) when every value is a canonical UUID. pa.binary(16) uses 16 raw bytes instead of a 36-character string. The writer picks the type per logical id and writes it into the Parquet schemas. The reader infers the types from the records file schema. binary(16) decodes back to the canonical string, so callers always see string ids.

IdTypes module-attribute source

IdTypes = dict[str, pa.DataType]

LOGICAL_IDS module-attribute source

LOGICAL_IDS: tuple[str, ...] = (
    "record_id",
    "time_series_id",
    "annotation_id",
    "task_id",
    "source_id",
    "subject_id",
)

TASK_COMMON_NAMES module-attribute source

TASK_COMMON_NAMES: tuple[str, ...] = (
    "id",
    "record_ids",
    "from_task_ids",
    "prompt",
    "scope",
    "input_annotation_ids",
    "target_annotation_ids",
    "rationale",
)

UUID16 module-attribute source

UUID16 = pa.binary(16)

IdCodec dataclass source

Convert ids between their in-memory strings and their on-disk form.

This is the one place that holds the logical-id-per-column mapping and the bytes <-> str conversion. The writer and the reader each build a codec (:meth:from_uuid16 or :meth:from_encoding). Both call it, so the two halves of the format contract cannot drift apart.

uuid16 instance-attribute source
uuid16: frozenset[str]

The logical ids whose columns are stored as binary(16). The rest are pa.string().

decode source
decode(logical: str, value: object) -> str

Decode one required id, turning 16 raw bytes back into a canonical string for uuid16.

Parameters:

Name Type Description Default
logical str

The logical id the column holds.

 required
value object

The raw cell value (bytes for a uuid16 column, string otherwise).

 required

Returns:

Type Description
str

The canonical id string.
decode_list source
decode_list(logical: str, values: object) -> list[str]

Decode a list of id values element-wise via :meth:decode.

Parameters:

Name Type Description Default
logical str

The logical id the column holds.

 required
values object

The raw cell values.

 required

Returns:

Type Description
list[str]

The decoded id strings.
decode_opt source
decode_opt(logical: str, value: object) -> str | None

Decode an optional id (for example, source_id), passing None through.

Parameters:

Name Type Description Default
logical str

The logical id the column holds.

 required
value object

The raw cell value, or None.

 required

Returns:

Type Description
str | None

The canonical id string, or None.
decode_payload source
decode_payload(
    refs: TaskRefs, name: str, value: object
) -> object

Decode a task payload cell holding ids or spans, leaving plain payload untouched.

Parameters:

Name Type Description Default
refs TaskRefs

The owning task class's reference declaration.

 required
name str

The payload column name.

 required
value object

The (already tuple-normalized) cell value.

 required

Returns:

Type Description
object

The decoded value.
decode_span source
decode_span(row: object) -> Span | None

Rebuild a span from its struct row, decoding the series ids it is scoped to.

Parameters:

Name Type Description Default
row object

The struct row read from a task partition, or None.

 required

Returns:

Type Description
Span | None

The span, or None.

Raises:

Type Description
TimeFFormatError

If the frame is neither "seconds" nor "steps", or a step span's stored id list is empty.
encode source
encode(logical: str, value: object) -> object

Encode one id to 16 raw bytes for a uuid16 column, else pass it through unchanged.

A reference id outside the entity id space (for example, a ForecastingTask.target_record_id that names no record) raises a contextual :class:TimeFValidationError naming the column and value. This makes the writer fail legibly even if referential validation is bypassed.

Parameters:

Name Type Description Default
logical str

The logical id the column holds (for example, "record_id").

 required
value object

The id string, or None.

 required

Returns:

Type Description
object

The 16-byte form for a uuid16 column, else value unchanged.

Raises:

Type Description
TimeFValidationError

If a uuid16 column holds a non-canonical-UUID id.
encode_list source
encode_list(logical: str, values: Iterable[object]) -> list

Encode a list of ids element-wise via :meth:encode.

Parameters:

Name Type Description Default
logical str

The logical id the column holds.

 required
values Iterable[object]

The id strings.

 required

Returns:

Type Description
list

The encoded list.
encode_payload source
encode_payload(
    refs: TaskRefs, name: str, value: object
) -> object

Encode a task payload cell holding ids or spans, leaving plain payload untouched.

Parameters:

Name Type Description Default
refs TaskRefs

The owning task class's reference declaration.

 required
name str

The payload column name.

 required
value object

The (already list-normalized) cell value.

 required

Returns:

Type Description
object

The encoded value.
encode_span source
encode_span(span: Span | None) -> dict | None

Encode a span to its struct row, encoding the series ids it is scoped to.

Parameters:

Name Type Description Default
span Span | None

The span, or None for a whole-record scope.

 required

Returns:

Type Description
dict | None

The struct row, or None.

Raises:

Type Description
TimeFValidationError

If span is not a concrete time or step span.
from_id_types classmethod source
from_id_types(id_types: IdTypes) -> IdCodec

Build a codec from resolved Arrow id types (the reader's entry point).

Parameters:

Name Type Description Default
id_types IdTypes

Mapping from logical id to its Arrow type.

 required

Returns:

Type Description
IdCodec

The codec for those columns.
from_uuid16 classmethod source
from_uuid16(uuid16: Iterable[str]) -> IdCodec

Build a codec from the set of logical ids stored as binary(16) (the writer's entry point).

Parameters:

Name Type Description Default
uuid16 Iterable[str]

The logical ids whose columns are binary(16).

 required

Returns:

Type Description
IdCodec

The codec for those columns.

annotations_schema source

annotations_schema(id_types: IdTypes) -> Schema

Return the annotations table schema.

Parameters:

Name Type Description Default
id_types IdTypes

The resolved id storage types.

 required

Returns:

Type Description
Schema

The Arrow schema.

default_id_types source

default_id_types() -> IdTypes

Return the all-string id types.

Returns:

Type Description
IdTypes

A mapping from every logical id to pa.string().

id_types_from_records_schema source

id_types_from_records_schema(schema: Schema) -> IdTypes

Infer every logical id's Arrow type from the records Parquet schema.

Parameters:

Name Type Description Default
schema Schema

The Arrow schema of any records part file.

 required

Returns:

Type Description
IdTypes

A mapping from every logical id to its Arrow type.

index_schema source

index_schema(id_types: IdTypes) -> Schema

Return the time-series index table schema.

Parameters:

Name Type Description Default
id_types IdTypes

The resolved id storage types.

 required

Returns:

Type Description
Schema

The Arrow schema.

records_schema source

records_schema(id_types: IdTypes) -> Schema

Return the records table schema.

Parameters:

Name Type Description Default
id_types IdTypes

The resolved id storage types.

 required

Returns:

Type Description
Schema

The Arrow schema.

shard_schema source

shard_schema(
    id_types: IdTypes,
    value_type: DataType = _DEFAULT_VALUES_TYPE,
) -> Schema

Return the waveform shard schema.

Parameters:

Name Type Description Default
id_types IdTypes

The resolved id storage types.

 required
value_type DataType

The element type of the values list (defaults to float32).

 _DEFAULT_VALUES_TYPE

Returns:

Type Description
Schema

The Arrow schema.

span_struct source

span_struct(id_types: IdTypes) -> DataType

Return the struct type a :class:~timenet.types.Span is stored as.

Parameters:

Name Type Description Default
id_types IdTypes

The resolved id storage types.

 required

Returns:

Type Description
DataType

The struct type used for a task's scope and for localization target spans.

task_schema source

task_schema(
    task_type: TaskType, id_types: IdTypes | None = None
) -> Schema

Return the Arrow schema for one task partition.

Parameters:

Name Type Description Default
task_type TaskType

The task type whose partition schema to build.

 required
id_types IdTypes | None

The resolved id storage types (defaults to all-string). The reader passes none, because it only reads the column names.

 None

Returns:

Type Description
Schema

The Arrow schema (common columns plus the type's payload columns).

Raises:

Type Description
TimeFValidationError

If the task dataclass payload and its Arrow schema have drifted.

time_series_struct source

time_series_struct(id_types: IdTypes) -> DataType

Return the per-record nested time-series struct type.

Parameters:

Name Type Description Default
id_types IdTypes

The resolved id storage types.

 required

Returns:

Type Description
DataType

The struct type used inside records.time_series.