TimeF format¶

TimeF stores one dataset version as one directory. The directory holds a JSON manifest, a few Parquet tables, and the time-series values. This page describes each file, the columns inside it, and how the parts link together. It also describes how the writer encodes the values.

The manifest is the contract. A reader loads manifest.json first and never runs connector code. Everything else is addressed from the manifest, not from a directory scan.

On-disk layout¶

A version directory looks like this. A registry addresses it by org/name and version. Its open_version() method returns a DatasetVersion handle for the version: the parsed manifest plus a filesystem-rooted view of its files.

<org>/<name>/<version>/
├── manifest.json       # the contract: metadata, schema, and the file list
├── records/            # one row per record (the control plane)
│   └── part-00000000.parquet
├── annotations/        # one row per annotation
│   └── part-00000000.parquet
├── time_series_index/  # locates every values chunk
│   └── part-00000000.parquet
├── tasks/              # one partition directory per task type
│   ├── task=classification/part-00000000.parquet
│   └── task=answer/part-00000000.parquet
└── time_series/        # the values plane (default Parquet backend)
    ├── part-00000000.parquet
    └── part-00000001.parquet

The reader does not glob these paths. The manifest's files block lists every part, so a table can shard into more parts later without a format change.

Two planes¶

TimeF splits a dataset into a control plane and a values plane.

The control plane is the record, annotation, task, and index tables. These are always Parquet, and they do not depend on how the values are stored.

The values plane is the typed waveform of every series. This is the one part whose storage is swappable. The manifest's values_backend field names the backend: parquet (the default, rotating shards) or zarr (a chunked array store). The control plane stays the same either way.

The manifest¶

manifest.json is a single JSON object. A reader parses it, checks timef_format_version against the versions it supports, and rebuilds the dataset types from the flat descriptors it holds.

The control-plane tables¶

Every table is Parquet. The column layout is fixed. Only the id columns change type, and the records table's Parquet schema carries that choice.

records/part-00000000.parquet¶

One row per record. A record groups the series that were recorded together, plus the tasks and annotations that point at them.

The time_series struct carries each series' identity and its time axis. It holds spec_type, signal, source_id, time_series_id, an axis_type, and n_values. A regular axis fills period_numerator_us, period_denominator, and start_index. An irregular axis fills first_time_offset_us and last_time_offset_us instead. The struct does not hold the values themselves.

annotations/part-00000000.parquet¶

One row per annotation.

A span struct holds start_us, end_us, and the time_series_ids it is scoped to. A null end_us means the span is a point at start_us.

time_series_index/part-00000000.parquet¶

One row per values chunk. This table is the join between a series and its bytes on disk. A series is split into chunks, and each chunk gets one row here.

The two locator columns are backend-neutral. For the Parquet backend, chunk_major_idx is the row group in the shard and chunk_minor_idx is the row within that row group. For the Zarr backend, chunk_major_idx is the element-start index in the per-modality array and chunk_minor_idx is null.

tasks/task=<type>/part-00000000.parquet¶

Tasks are partitioned by type, one directory per task type. Every partition shares a common set of columns: id, record_ids, from_task_ids, prompt, scope, input_annotation_ids, target_annotation_ids, and rationale. Each type then adds its own payload columns. A classification task adds target and target_schema. A scalar prediction adds target, unit, and target_name. A temporal localization adds a list of span structs.

The values plane¶

The waveform values live outside the record table, in the values plane. The default backend writes rotating Parquet shards.

part-00000000.parquet¶

The writer builds the shards in a fixed order. It dedupes series by time_series_id, sorts them by (spec_type, signal, time_series_id), and streams them through the backend. It splits each series into chunks of at most chunk_max_bytes, buffers chunks until they reach row_group_target_bytes, and flushes them as one row group. A shard rotates once it reaches shard_target_bytes. A row group never spans two shards, so the index locators are exact.

The defaults are a 128 MiB shard target, a 4 MiB row-group target, a 1 MiB chunk limit, and zstd compression at level 19 (Parquet) or 9 (Zarr).

How the parts link together¶

Ids tie the whole dataset together. Six logical ids cross-reference the entities: record_id, time_series_id, annotation_id, task_id, source_id, and subject_id. A record lists the ids of its series, tasks, annotations, and subjects. A task and an annotation each list the record ids they point at.

To read one series, the reader joins the record to its bytes through the index:

1. Read the record row and take a time_series_id from its time_series list.
2. Find that series' chunks in the time_series_index table.
3. For each chunk, open chunk_file and go to chunk_major_idx, then chunk_minor_idx.
4. Read the values list and rebuild the series on its time axis.

The reader loads the manifest, tasks, annotations, and index eagerly. It keeps the values and the record objects lazy, so a large dataset opens without reading every shard.

Encodings¶

The writer pins each column's Parquet encoding by its data role, rather than leaving the choice to pyarrow. Pinned encodings keep a re-built version byte-stable.

Every file also carries column statistics, a page index, and per-page checksums. Every file uses content-defined chunking, which aligns data pages to content. A re-built or edited version then re-stores only the pages that changed on a deduplicating backend such as Xet.

Choosing the values encoding¶

No single encoding is best for every waveform, so the writer measures the data instead of pinning one. It samples the values it has already buffered for a modality, counts the distinct values (bit patterns for floats), and picks:

• dictionary at or below 65,536 distinct values,
• BYTE_STREAM_SPLIT above that for floats,
• plain above that for strings and integers (byte-plane splitting has no meaning for these types).

Bool signals always use plain and enum signals always use dictionary, both without measuring anything. The writer takes one decision per spec_type, before that modality's first shard opens. The decision reads only buffered data, so re-building an unchanged source reaches the same encoding and writes the same bytes.

The rule follows the measurements. On real data, at zstd level 3, the values column measures:

BYTE_STREAM_SPLIT transposes each float into four byte planes and compresses each plane apart. It wins on smooth, high-cardinality signals, where the sign and high-mantissa planes are near constant. It loses on quantized data, where the low mantissa byte is noise the split isolates into an incompressible plane. Dictionary wins there, because a physical conversion onto a fixed grid (a 0.001 mV step, an integer ADC scale) leaves only a few thousand distinct values behind tens of millions of samples.

The writer records its choice in the manifest under value_encoding, a spec_type to encoding map. That is provenance, not contract. Parquet already records the applied encoding in every file's footer, so a reader decodes a shard without the manifest.

One case needs a manual override: high-cardinality quantized data (say a 24-bit integer-scaled signal) suits neither branch. To force an encoding, set value_encoding on the dataset card, or pass it to the writer:

with TimeFWriter(root, dataset, value_encoding="dictionary") as writer:
    writer.write()

Id storage¶

An entity id is a UUIDv7 string by default. When every value in an id's space is a canonical UUID, the writer stores that column as 16 raw bytes (binary(16)) instead of a 36-character string. A reader reads the choice off the records table's Parquet schema and decodes the bytes back to the canonical string, so a caller always sees a string id.

Integrity¶

The manifest lists a sha256: checksum for every file. The writer hashes each file a block at a time as it commits the version. A reader can hash the files again and compare, so a truncated or corrupt download fails loudly instead of returning wrong data.

Related pages¶

• TimeFWriter covers the write path and its options.
• TimeFReader covers the read path.
• Manifest documents every manifest field in detail.