# Format

## timenet.format

The TimeF on-disk format contract: filenames, layout templates, and pinned Arrow schemas.

The writer and the reader both use this module. They are otherwise independent. The definitions that
both sides of the round-trip must agree on live here, so neither side imports the other.

### checksums

File checksums recorded in the manifest and verified on read.

The reader must hash files the same way the writer did, or :meth:`~timenet.reader.TimeFReader.verify`
means nothing. The algorithm and the block size live here so both sides call the same code.

#### CHECKSUM\_PREFIX `module-attribute`

```
CHECKSUM_PREFIX = 'sha256:'
```

#### file\_checksum

```
file_checksum(path: Path) -> str
```

Return a file's manifest checksum, hashing it a block at a time.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `path` | `Path` | The file to hash. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `str` | The checksum as `"sha256:<hex>"`. |

#### stream\_checksum

```
stream_checksum(handle: _Readable) -> str
```

Return an open binary stream's manifest checksum, hashing it a block at a time.

This is the stream counterpart of :func:`file_checksum`. A reader can use it to hash a file
opened through a pyarrow filesystem (local now, an object store later) instead of a local
`Path`.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `handle` | `_Readable` | An open binary stream positioned at the start. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `str` | The checksum as `"sha256:<hex>"`. |

### constants

On-disk filenames and layout templates for the TimeF format.

#### ANNOTATIONS\_SORT\_KEY `module-attribute`

```
ANNOTATIONS_SORT_KEY = 'id'
```

#### ANNOTATIONS\_TEMPLATE `module-attribute`

```
ANNOTATIONS_TEMPLATE = 'annotations/part-{:08d}.parquet'
```

#### DEFAULT\_CHUNK\_MAX\_BYTES `module-attribute`

```
DEFAULT_CHUNK_MAX_BYTES = 1 * 2 ** 20
```

#### DEFAULT\_COMPRESSION `module-attribute`

```
DEFAULT_COMPRESSION = 'zstd'
```

#### DEFAULT\_CONTROL\_SHARD\_TARGET\_BYTES `module-attribute`

```
DEFAULT_CONTROL_SHARD_TARGET_BYTES = 128 * 2 ** 20
```

#### DEFAULT\_ROW\_GROUP\_TARGET\_BYTES `module-attribute`

```
DEFAULT_ROW_GROUP_TARGET_BYTES = 4 * 2 ** 20
```

#### DEFAULT\_SHARD\_TARGET\_BYTES `module-attribute`

```
DEFAULT_SHARD_TARGET_BYTES = 128 * 2 ** 20
```

#### INDEX\_SORT\_KEY `module-attribute`

```
INDEX_SORT_KEY = 'record_id'
```

#### INDEX\_TEMPLATE `module-attribute`

```
INDEX_TEMPLATE = 'time_series_index/part-{:08d}.parquet'
```

#### MANIFEST\_FILE `module-attribute`

```
MANIFEST_FILE = 'manifest.json'
```

#### MAX\_PART\_INDEX `module-attribute`

```
MAX_PART_INDEX = 10 ** PART_INDEX_DIGITS - 1
```

#### PART\_INDEX\_DIGITS `module-attribute`

```
PART_INDEX_DIGITS = 8
```

#### RECORDS\_TEMPLATE `module-attribute`

```
RECORDS_TEMPLATE = 'records/part-{:08d}.parquet'
```

#### SHARD\_DIR `module-attribute`

```
SHARD_DIR = 'time_series'
```

#### SHARD\_TEMPLATE `module-attribute`

```
SHARD_TEMPLATE = 'time_series/part-{:08d}.parquet'
```

#### TASKS\_DIR `module-attribute`

```
TASKS_DIR = 'tasks'
```

#### TASK\_PART\_TEMPLATE `module-attribute`

```
TASK_PART_TEMPLATE = (
    "tasks/task={task_type}/part-{:08d}.parquet"
)
```

#### check\_relative\_path

```
check_relative_path(name: str, path: str) -> None
```

Reject a path that would resolve outside the dataset root it is meant to stay under.

Callers join a manifest file entry and a registry handle's `relpath` onto a root directory
(or filesystem prefix) as-is. An absolute path or a `..` segment would escape that root instead
of raising. This would let a crafted manifest or caller read or write outside the intended
directory.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `name` | `str` | The name of the field to check. It appears in the error message. | *required* |
| `path` | `str` | The path to check. It must be relative and rooted inside the dataset. | *required* |

Raises:

| Type | Description |
| --- | --- |
| `TimeFValidationError` | If `path` is absolute, or has a `..` segment. |

#### part\_path

```
part_path(template: str, index: int, **fields: str) -> str
```

Render a numbered part/shard path, refusing an index that would overflow the fixed width.

The reader visits parts in the order the manifest lists them. The zero-padded names keep an
`ls` or a prefix listing in that same order. That order only holds while every number fits in
`PART_INDEX_DIGITS`. A ninth digit sorts before the eighth and silently breaks it. Routing
every part name through here turns that overflow into a loud error instead.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `template` | `str` | A layout template whose part number is the `{:08d}` field, for example `SHARD_TEMPLATE`. | *required* |
| `index` | `int` | The zero-based part number to render. | *required* |
| `fields` | `str` | Any remaining named fields the template needs, for example `task_type`. | `{}` |

Returns:

| Type | Description |
| --- | --- |
| `str` | The formatted version-relative path. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFValidationError` | If `index` is negative or needs more than `PART_INDEX_DIGITS` digits. |

### schemas

Arrow schemas for the TimeF on-disk files, parameterized by id storage type.

The column layout is fixed. Only the id columns vary. Every id column holds one of the six logical ids
in :data:`LOGICAL_IDS`. Each is stored as `pa.string()`, or as `pa.binary(16)` when every value is a
canonical UUID. `pa.binary(16)` uses 16 raw bytes instead of a 36-character string. The writer picks
the type per logical id and writes it into the Parquet schemas. The reader infers the types from the
records file schema. `binary(16)` decodes back to the canonical string, so callers always see
string ids.

#### IdTypes `module-attribute`

```
IdTypes = dict[str, pa.DataType]
```

#### LOGICAL\_IDS `module-attribute`

```
LOGICAL_IDS: tuple[str, ...] = (
    "record_id",
    "time_series_id",
    "annotation_id",
    "task_id",
    "source_id",
    "subject_id",
)
```

#### TASK\_COMMON\_NAMES `module-attribute`

```
TASK_COMMON_NAMES: tuple[str, ...] = (
    "id",
    "record_ids",
    "from_task_ids",
    "prompt",
    "scope",
    "input_annotation_ids",
    "target_annotation_ids",
    "rationale",
)
```

#### UUID16 `module-attribute`

```
UUID16 = pa.binary(16)
```

#### IdCodec `dataclass`

Convert ids between their in-memory strings and their on-disk form.

This is the one place that holds the logical-id-per-column mapping and the `bytes <-> str`
conversion. The writer and the reader each build a codec (:meth:`from_uuid16` or
:meth:`from_encoding`). Both call it, so the two halves of the format contract cannot drift apart.

##### uuid16 `instance-attribute`

```
uuid16: frozenset[str]
```

The logical ids whose columns are stored as `binary(16)`. The rest are `pa.string()`.

##### decode

```
decode(logical: str, value: object) -> str
```

Decode one required id, turning 16 raw bytes back into a canonical string for `uuid16`.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `logical` | `str` | The logical id the column holds. | *required* |
| `value` | `object` | The raw cell value (bytes for a `uuid16` column, string otherwise). | *required* |

Returns:

| Type | Description |
| --- | --- |
| `str` | The canonical id string. |

##### decode\_list

```
decode_list(logical: str, values: object) -> list[str]
```

Decode a list of id values element-wise via :meth:`decode`.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `logical` | `str` | The logical id the column holds. | *required* |
| `values` | `object` | The raw cell values. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `list[str]` | The decoded id strings. |

##### decode\_opt

```
decode_opt(logical: str, value: object) -> str | None
```

Decode an optional id (for example, `source_id`), passing `None` through.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `logical` | `str` | The logical id the column holds. | *required* |
| `value` | `object` | The raw cell value, or `None`. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `str | None` | The canonical id string, or `None`. |

##### decode\_payload

```
decode_payload(
    refs: TaskRefs, name: str, value: object
) -> object
```

Decode a task payload cell holding ids or spans, leaving plain payload untouched.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `refs` | `TaskRefs` | The owning task class's reference declaration. | *required* |
| `name` | `str` | The payload column name. | *required* |
| `value` | `object` | The (already tuple-normalized) cell value. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `object` | The decoded value. |

##### decode\_span

```
decode_span(row: object) -> Span | None
```

Rebuild a span from its struct row, decoding the series ids it is scoped to.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `row` | `object` | The struct row read from a task partition, or `None`. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `Span | None` | The span, or `None`. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFFormatError` | If the frame is neither `"seconds"` nor `"steps"`, or a step span's stored id list is empty. |

##### encode

```
encode(logical: str, value: object) -> object
```

Encode one id to 16 raw bytes for a `uuid16` column, else pass it through unchanged.

A reference id outside the entity id space (for example, a `ForecastingTask.target_record_id`
that names no record) raises a contextual :class:`TimeFValidationError` naming the column and
value. This makes the writer fail legibly even if referential validation is bypassed.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `logical` | `str` | The logical id the column holds (for example, `"record_id"`). | *required* |
| `value` | `object` | The id string, or `None`. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `object` | The 16-byte form for a `uuid16` column, else `value` unchanged. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFValidationError` | If a `uuid16` column holds a non-canonical-UUID id. |

##### encode\_list

```
encode_list(logical: str, values: Iterable[object]) -> list
```

Encode a list of ids element-wise via :meth:`encode`.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `logical` | `str` | The logical id the column holds. | *required* |
| `values` | `Iterable[object]` | The id strings. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `list` | The encoded list. |

##### encode\_payload

```
encode_payload(
    refs: TaskRefs, name: str, value: object
) -> object
```

Encode a task payload cell holding ids or spans, leaving plain payload untouched.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `refs` | `TaskRefs` | The owning task class's reference declaration. | *required* |
| `name` | `str` | The payload column name. | *required* |
| `value` | `object` | The (already list-normalized) cell value. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `object` | The encoded value. |

##### encode\_span

```
encode_span(span: Span | None) -> dict | None
```

Encode a span to its struct row, encoding the series ids it is scoped to.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `span` | `Span | None` | The span, or `None` for a whole-record scope. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `dict | None` | The struct row, or `None`. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFValidationError` | If `span` is not a concrete time or step span. |

##### from\_id\_types `classmethod`

```
from_id_types(id_types: IdTypes) -> IdCodec
```

Build a codec from resolved Arrow id types (the reader's entry point).

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | Mapping from logical id to its Arrow type. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `IdCodec` | The codec for those columns. |

##### from\_uuid16 `classmethod`

```
from_uuid16(uuid16: Iterable[str]) -> IdCodec
```

Build a codec from the set of logical ids stored as `binary(16)` (the writer's entry point).

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `uuid16` | `Iterable[str]` | The logical ids whose columns are `binary(16)`. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `IdCodec` | The codec for those columns. |

#### annotations\_schema

```
annotations_schema(id_types: IdTypes) -> Schema
```

Return the annotations table schema.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | The resolved id storage types. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `Schema` | The Arrow schema. |

#### default\_id\_types

```
default_id_types() -> IdTypes
```

Return the all-`string` id types.

Returns:

| Type | Description |
| --- | --- |
| `IdTypes` | A mapping from every logical id to `pa.string()`. |

#### id\_types\_from\_records\_schema

```
id_types_from_records_schema(schema: Schema) -> IdTypes
```

Infer every logical id's Arrow type from the records Parquet schema.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `schema` | `Schema` | The Arrow schema of any records part file. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `IdTypes` | A mapping from every logical id to its Arrow type. |

#### index\_schema

```
index_schema(id_types: IdTypes) -> Schema
```

Return the time-series index table schema.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | The resolved id storage types. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `Schema` | The Arrow schema. |

#### records\_schema

```
records_schema(id_types: IdTypes) -> Schema
```

Return the records table schema.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | The resolved id storage types. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `Schema` | The Arrow schema. |

#### shard\_schema

```
shard_schema(
    id_types: IdTypes,
    value_type: DataType = _DEFAULT_VALUES_TYPE,
) -> Schema
```

Return the waveform shard schema.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | The resolved id storage types. | *required* |
| `value_type` | `DataType` | The element type of the `values` list (defaults to `float32`). | `_DEFAULT_VALUES_TYPE` |

Returns:

| Type | Description |
| --- | --- |
| `Schema` | The Arrow schema. |

#### span\_struct

```
span_struct(id_types: IdTypes) -> DataType
```

Return the struct type a :class:`~timenet.types.Span` is stored as.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | The resolved id storage types. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `DataType` | The struct type used for a task's `scope` and for localization target spans. |

#### task\_schema

```
task_schema(
    task_type: TaskType, id_types: IdTypes | None = None
) -> Schema
```

Return the Arrow schema for one task partition.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `task_type` | `TaskType` | The task type whose partition schema to build. | *required* |
| `id_types` | `IdTypes | None` | The resolved id storage types (defaults to all-`string`). The reader passes none, because it only reads the column names. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `Schema` | The Arrow schema (common columns plus the type's payload columns). |

Raises:

| Type | Description |
| --- | --- |
| `TimeFValidationError` | If the task dataclass payload and its Arrow schema have drifted. |

#### time\_series\_struct

```
time_series_struct(id_types: IdTypes) -> DataType
```

Return the per-record nested time-series struct type.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `id_types` | `IdTypes` | The resolved id storage types. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `DataType` | The struct type used inside `records.time_series`. |