Format

timenet.format ¶source

The TimeF on-disk format contract: filenames, layout templates, and pinned Arrow schemas.

The writer and the reader both use this module. They are otherwise independent. The definitions that both sides of the round-trip must agree on live here, so neither side imports the other.

annotation_codec ¶source

Codec between an annotation's value and the typed value columns of annotation_contents.

An annotation value is a string, an integer, a float, a boolean, a list of strings, or absent for a pure marker. Each shape has its own column so DuckDB can filter on it and the reader hands back the exact Python type that was written: an integer stays an integer and a boolean stays a boolean.

VALUE_COLUMNS `module-attribute` ¶source

VALUE_COLUMNS = (
    "value_kind",
    "text_value",
    "integer_value",
    "float_value",
    "boolean_value",
    "text_list_value",
)

The annotation_contents columns that together hold one value.

VALUE_KINDS `module-attribute` ¶source

VALUE_KINDS = tuple(
    (kind.value) for kind in AnnotationValueKind
)

Every value_kind a stored annotation may carry. A marker has NULL.

decode_annotation_value ¶source

decode_annotation_value(
    value_kind: str | None,
    text_value: str | None,
    integer_value: int | None,
    float_value: float | None,
    boolean_value: bool | None,
    text_list_value: list[str] | None,
) -> Any

Rebuild one annotation value from its columns.

Returns:

Type	Description
`Any`	The value in the Python type it was written with, or `None` for a marker.

Raises:

Type	Description
`TimeFFormatError`	If the kind is unknown or its column is NULL.

encode_annotation_value ¶source

encode_annotation_value(
    value: object, *, label: str
) -> dict[str, Any]

Project one annotation value onto the value columns.

Parameters:

Name	Type	Description	Default
`value`	`object`	The annotation's value, or `None` for a marker.	required
`label`	`str`	How to name the annotation in an error message.	required

Returns:

Type	Description
`dict[str, Any]`	One entry per :data:`VALUE_COLUMNS`, all `None` for a marker.

Raises:

Type	Description
`TimeFValidationError`	If the value is not one of the supported shapes.

annotation_value_kind ¶source

Tags for annotation values stored in the control catalog.

AnnotationValueKind ¶source

Bases: StrEnum

Identify the populated value column of an annotation's content.

BOOLEAN `class-attribute` `instance-attribute` ¶source

BOOLEAN = 'boolean'

FLOAT `class-attribute` `instance-attribute` ¶source

FLOAT = 'float'

INTEGER `class-attribute` `instance-attribute` ¶source

INTEGER = 'integer'

TEXT `class-attribute` `instance-attribute` ¶source

TEXT = 'text'

TEXT_LIST `class-attribute` `instance-attribute` ¶source

TEXT_LIST = 'text_list'

checksums ¶source

File checksums recorded in the manifest and verified on read.

The reader must hash files the same way the writer did, or :meth:~timenet.reader.TimeFReader.verify means nothing. The algorithm and the block size live here so both sides call the same code.

CHECKSUM_PREFIX `module-attribute` ¶source

CHECKSUM_PREFIX = 'sha256:'

file_checksum ¶source

file_checksum(path: Path) -> str

Return a file's manifest checksum, hashing it a block at a time.

Parameters:

Name	Type	Description	Default
`path`	`Path`	The file to hash.	required

Returns:

Type	Description
`str`	The checksum as `"sha256:<hex>"`.

stream_checksum ¶source

stream_checksum(handle: _Readable) -> str

Return an open binary stream's manifest checksum, hashing it a block at a time.

This is the stream counterpart of :func:file_checksum. A reader can use it to hash a file opened through a pyarrow filesystem (local now, an object store later) instead of a local Path.

Parameters:

Name	Type	Description	Default
`handle`	`_Readable`	An open binary stream positioned at the start.	required

Returns:

Type	Description
`str`	The checksum as `"sha256:<hex>"`.

constants ¶source

On-disk filenames and layout templates for the TimeF format.

DEFAULT_CHUNK_MAX_BYTES `module-attribute` ¶source

DEFAULT_CHUNK_MAX_BYTES = 1 * 2 ** 20

DEFAULT_COMPRESSION `module-attribute` ¶source

DEFAULT_COMPRESSION = 'zstd'

DEFAULT_ROW_GROUP_TARGET_BYTES `module-attribute` ¶source

DEFAULT_ROW_GROUP_TARGET_BYTES = 4 * 2 ** 20

DEFAULT_SHARD_TARGET_BYTES `module-attribute` ¶source

DEFAULT_SHARD_TARGET_BYTES = 128 * 2 ** 20

MANIFEST_FILE `module-attribute` ¶source

MANIFEST_FILE = 'manifest.json'

MAX_PART_INDEX `module-attribute` ¶source

MAX_PART_INDEX = 10 ** PART_INDEX_DIGITS - 1

PART_INDEX_DIGITS `module-attribute` ¶source

PART_INDEX_DIGITS = 8

PAYLOAD_TEMPLATE `module-attribute` ¶source

PAYLOAD_TEMPLATE = 'payloads/part-{:08d}.parquet'

SHARD_DIR `module-attribute` ¶source

SHARD_DIR = 'time_series'

SHARD_TEMPLATE `module-attribute` ¶source

SHARD_TEMPLATE = 'time_series/part-{:08d}.parquet'

check_relative_path ¶source

check_relative_path(name: str, path: str) -> str

Reject a path that would resolve outside the dataset root it is meant to stay under.

Callers join a manifest file entry and a registry handle's relpath onto a root directory (or filesystem prefix) as-is. An absolute path or a .. segment would escape that root instead of raising. This would let a crafted manifest or caller read or write outside the intended directory. Stored paths use forward slashes on every platform; Windows drive paths and backslashes are rejected even when validation runs on another operating system.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the field to check. It appears in the error message.	required
`path`	`str`	The path to check. It must be relative and rooted inside the dataset.	required

Returns:

Type	Description
`str`	The validated path.

Raises:

Type	Description
`TimeFValidationError`	If `path` is absolute, names a Windows drive, contains backslashes, or has a `..` segment.

part_path ¶source

part_path(template: str, index: int, **fields: str) -> str

Render a numbered part/shard path, refusing an index that would overflow the fixed width.

The reader visits parts in the order the manifest lists them. The zero-padded names keep an ls or a prefix listing in that same order. That order only holds while every number fits in PART_INDEX_DIGITS. A ninth digit sorts before the eighth and silently breaks it. Routing every part name through here turns that overflow into a loud error instead.

Parameters:

Name	Type	Description	Default
`template`	`str`	A layout template whose part number is the `{:08d}` field, for example `SHARD_TEMPLATE`.	required
`index`	`int`	The zero-based part number to render.	required
`fields`	`str`	Any remaining named fields the template needs, for example `task_type`.	`{}`

Returns:

Type	Description
`str`	The formatted version-relative path.

Raises:

Type	Description
`TimeFValidationError`	If `index` is negative or needs more than `PART_INDEX_DIGITS` digits.

control_audit ¶source

Audit a finished control.duckdb for relational integrity.

The control database has no persisted constraints. ART indexes on large tables such as signal_chunks make every write slower and the file larger. Instead, the writer keeps the invariants while it streams rows. It checks them in Python with ID sets. This module holds the SQL form of the same invariants. It scans every table, so it is not part of the write path. Tests run it after each round trip, and tooling can run it on a published version.

CROSS_RECORD_PARENT `module-attribute` ¶source

CROSS_RECORD_PARENT = "\n    SELECT child.source_id\n    FROM sources child\n    JOIN sources parent ON parent.source_id = child.parent_source_id\n    WHERE child.record_id <> parent.record_id\n    LIMIT 1\n"

SOURCE_CYCLE `module-attribute` ¶source

SOURCE_CYCLE = "\n    WITH RECURSIVE ancestors(source_id, parent_source_id, path, cyclic) AS (\n        SELECT source_id, parent_source_id, [source_id], false FROM sources\n        UNION ALL\n        SELECT parent.source_id, parent.parent_source_id,\n               list_append(ancestors.path, parent.source_id),\n               list_contains(ancestors.path, parent.source_id)\n        FROM ancestors\n        JOIN sources parent ON parent.source_id = ancestors.parent_source_id\n        WHERE NOT ancestors.cyclic\n    )\n    SELECT source_id FROM ancestors WHERE cyclic LIMIT 1\n"

audit_control_database ¶source

audit_control_database(
    connection: DuckDBPyConnection,
    *,
    require_chunks: bool = True,
) -> None

Scan every control table and raise an error at the first broken rule.

Parameters:

Name	Type	Description	Default
`connection`	`DuckDBPyConnection`	An open connection to the control database.	required
`require_chunks`	`bool`	Whether every Signal must be covered exactly by `signal_chunks` rows. A metadata-only database written without value placements has none.	`True`

Raises:

Type	Description
`TimeFFormatError`	If a key repeats, a reference points to a missing object, or a value is not allowed. Also if Sources do not form separate trees for each Record. Also if an axis or chunk length differs from its Signal. Also if classification labels break a rule of their label set.

control_cache ¶source

Materialize a verified DuckDB control file behind a local-path seam.

materialize_control ¶source

materialize_control(version: DatasetVersion) -> Path

Return a local path for a version's verified DuckDB control database.

Local versions already satisfy DuckDB's path requirement. For a remote filesystem, this function downloads only control.duckdb into a checksum-keyed cache. A future remote-attach implementation can replace this function without changing the schema or query layer.

Returns:

Type	Description
`Path`	A local path suitable for a read-only DuckDB connection.

Raises:

Type	Description
`TimeFFormatError`	If the manifest lacks a control file or the downloaded file fails verification.

control_reader ¶source

Build the TimeF object hierarchy from control.duckdb.

The objects hold their stored values behind value handles, unloaded.

RECORD_BATCH_SIZE `module-attribute` ¶source

RECORD_BATCH_SIZE = 1000

The most Records one batch of a Record stream holds, unless the stream sets another batch size.

TASK_BATCH_SIZE `module-attribute` ¶source

TASK_BATCH_SIZE = 5000

The most tasks one batch of a task read or stream holds, unless a stream sets another batch size.

ValueLoaderFactory `module-attribute` ¶source

ValueLoaderFactory = Callable[
    [int, TimeSeriesSpec, int], ArrayHandle
]

DuckDBControlReader ¶source

Keep one read-only DuckDB connection and build requested records from it.

The connection opens again on the next query after :meth:close, and after the process forks, since every connection closes before a fork.

external `instance-attribute` ¶source

external = external

path `instance-attribute` ¶source

path = Path(path)

chunk_location ¶source

chunk_location(signal_id: int) -> ChunkLocation

Return where one Signal's chunks are stored, from its catalog rows.

Raises:

Type	Description
`TimeFFormatError`	If the Signal has no stored chunks.

chunk_locations_by_ids ¶source

chunk_locations_by_ids(
    signal_ids: Iterable[int],
) -> dict[int, ChunkLocation | None]

Return where a batch of Signals' chunks are stored, from their catalog rows.

Parameters:

Name	Type	Description	Default
`signal_ids`	`Iterable[int]`	Signal IDs.	required

Returns:

Type	Description
`dict[int, ChunkLocation \| None]`	Locations keyed by Signal ID, each with the Signal's chunks in chunk-index order, or
`dict[int, ChunkLocation \| None]`	`None` for a Signal without stored chunks.

Raises:

Type	Description
`TimeFFormatError`	If one Signal's chunks are stored by more than one release.

close ¶source

close() -> None

Close the reader's DuckDB connection and drop its cached catalog facts.

import_map ¶source

import_map() -> tuple[dict[str, str], ImportMap]

Return the parent release of each parent alias, and the import map of the catalog.

Returns:

Type	Description
`dict[str, str]`	The parent release that `imports` names for each parent alias, and the local ID of each
`ImportMap`	object in `imported_objects`, keyed by parent alias, kind, and parent ID.

Raises:

Type	Description
`TimeFFormatError`	If the catalog lists a parent alias, or a parent ID under one alias and kind, more than once.

imported_ids ¶source

imported_ids(
    alias: str,
    kind: ObjectKind,
    parent_ids: Iterable[EntityId],
) -> tuple[str | None, dict[EntityId, EntityId]]

Return the parent release of a parent alias and the local IDs of objects imported under it.

Parameters:

Name	Type	Description	Default
`alias`	`str`	The parent alias the objects were imported under.	required
`kind`	`ObjectKind`	The kind of the objects.	required
`parent_ids`	`Iterable[EntityId]`	The objects' IDs in the parent release.	required

Returns:

Type	Description
`str \| None`	The parent release the catalog names for the alias, or `None` if it names none, and
`dict[EntityId, EntityId]`	the local object ID of each parent ID the catalog knows.

Raises:

Type	Description
`TimeFFormatError`	If the catalog lists the alias more than once, or lists one of the parent IDs more than once under the alias and kind.

iter_record_batches ¶source

iter_record_batches(
    record_ids: Iterable[int] | None = None,
    *,
    with_annotations: bool = True,
    batch_size: int = RECORD_BATCH_SIZE,
) -> Iterator[tuple[Record, ...]]

Yield complete records one bounded batch at a time, and build each batch on its own.

Without record_ids, the stream reads the Record IDs one page at a time, in read order. A page holds up to batch_size IDs. The stream builds one batch from each page that is not empty. The next page starts after the last read order of the page before. The stream stops after the first page that holds fewer than batch_size IDs. So the stream does not read all Record IDs before the first batch.

With record_ids, the stream checks every requested ID before it builds the first batch. Each batch reads only its own rows. The records of a batch share no object with another batch.

Parameters:

Name	Type	Description	Default
`record_ids`	`Iterable[int] \| None`	Requested IDs in result order, or `None` for every record in read order.	`None`
`with_annotations`	`bool`	Whether to build annotations on records, sources, and signals.	`True`
`batch_size`	`int`	The most records that a batch holds. By default, a batch holds up to 1,000 records (`RECORD_BATCH_SIZE`).	`RECORD_BATCH_SIZE`

Yields:

Type	Description
`tuple[Record, ...]`	The records of each batch, with their stored values behind unloaded handles.

Raises:

Type	Description
`TimeFValidationError`	If `batch_size` is not a positive int, or if a requested ID is not an int.
`TimeFNotFoundError`	If a requested record is absent.

iter_task_batches ¶source

iter_task_batches(
    records: Iterable[Record] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
    batch_size: int = TASK_BATCH_SIZE,
) -> Iterator[list[Task]]

Yield the tasks whose inputs include one of records, a bounded batch at a time.

Each batch runs a handful of keyed queries and builds only that batch's tasks, so a caller that walks one Record at a time never holds the whole task table in memory. Only None skips the input Record filter; an empty collection selects no tasks.

Each batch is built on its own. The parents its tasks derive from through from_tasks, and the Records, Signals, and annotations they refer to outside records, are built for the batch and shared only among its tasks, so nothing of a yielded batch stays with the stream.

Parameters:

Name	Type	Description	Default
`records`	`Iterable[Record] \| None`	Input Records to select, or `None` for every task. Tasks reuse these objects.	`None`
`split`	`Split \| str \| None`	Keep only tasks of this split, or `None` for every task.	`None`
`batch_size`	`int`	The most tasks a batch holds. By default, a batch holds up to 5,000 tasks (`TASK_BATCH_SIZE`). Pass another value for larger or smaller batches.	`TASK_BATCH_SIZE`

Yields:

Type	Description
`list[Task]`	Each batch's concrete tasks, in stable ID order across batches.

Raises:

Type	Description
`TimeFFormatError`	If a relationship refers to a missing object or has an invalid field.

read_annotation_table ¶source

read_annotation_table(
    object_type: ObjectKind | None = None,
) -> Table

Return every annotation occurrence joined with its content as Arrow.

Parameters:

Name	Type	Description	Default
`object_type`	`ObjectKind \| None`	Keep only occurrences on this object type, or `None` for all.	`None`

Returns:

Type	Description
`Table`	One row per occurrence. Each row has the annotated object's type and ID, the content's
`Table`	ID and key, typed value columns, unit, span columns, confidence, and JSON metadata. The
`Table`	object ID is NULL for the Dataset, which has a string `dataset_id` instead.

Raises:

Type	Description
`TimeFFormatError`	If `object_type` is not an annotated object type.

read_dataset_annotations ¶source

read_dataset_annotations(
    dataset_id: str,
    *,
    by_occurrence: _AnnotationsByOccurrence | None = None,
) -> tuple[Annotation, ...]

Return annotations attached directly to a dataset, as new objects on every call.

Parameters:

Name	Type	Description	Default
`dataset_id`	`str`	The dataset whose annotations to read.	required
`by_occurrence`	`_AnnotationsByOccurrence \| None`	Filled with every annotation by its occurrence id, so that a task read of the same call can reference these objects.	`None`

Returns:

Type	Description
`tuple[Annotation, ...]`	Dataset-level annotation occurrences in storage order.

read_last_ids ¶source

read_last_ids() -> dict[ObjectKind, int]

Return the last used id per entity kind, as the writer stored it.

Returns:

Type	Description
`dict[ObjectKind, int]`	The stored counters. The result does not include a kind without a stored value.

Raises:

Type	Description
`TimeFFormatError`	If a stored counter names an unknown kind or is not an integer.

read_records ¶source

read_records(
    record_ids: Iterable[int] | None = None,
    *,
    with_annotations: bool = True,
    shared_offsets: dict[EntityId, StoredOffsets]
    | None = None,
) -> tuple[Record, ...]

Build complete recursive records, with their stored values behind unloaded handles.

Source links, clocks, axes, and time-offset locations are checked before any object is built. No payload file is opened.

Parameters:

Name	Type	Description	Default
`record_ids`	`Iterable[int] \| None`	Requested IDs in result order, or `None` for every record in read order.	`None`
`with_annotations`	`bool`	Whether to build annotations on records, sources, and signals.	`True`
`shared_offsets`	`dict[EntityId, StoredOffsets] \| None`	Time-offset handles by axis ID, shared with earlier record reads of the same call, and extended with this read's new ones. `None` shares nothing.	`None`

Returns:

Type	Description
`tuple[Record, ...]`	The built records.

Raises:

Type	Description
`TimeFValidationError`	If a requested ID is not an int.
`TimeFNotFoundError`	If a requested record is absent.
`TimeFFormatError`	If the stored hierarchy is inconsistent.

read_signals_by_id ¶source

read_signals_by_id(
    signal_ids: Collection[EntityId],
) -> dict[EntityId, Signal]

Build the Records that hold the given Signals, and return those Signals.

Each Record is built once, however many of its Signals are asked for, and the Records of one call share one time-offsets handle per axis.

Parameters:

Name	Type	Description	Default
`signal_ids`	`Collection[EntityId]`	The Signals' IDs.	required

Returns:

Type	Description
`dict[EntityId, Signal]`	The Signals by ID, each inside its Record, with stored values behind unloaded handles.

Raises:

Type	Description
`TimeFNotFoundError`	If an ID names no stored Signal.

read_target_table ¶source

read_target_table(
    record_ids: Iterable[int] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> Table

Return every inline target as Arrow, one typed row each, with references as public IDs.

The columns follow :class:timenet.format.task_target_row.TargetValueRow: a step span names its one series in span_signal_ids.

Parameters:

Name	Type	Description	Default
`record_ids`	`Iterable[int] \| None`	Keep only targets of tasks whose inputs include one of these Records.	`None`
`split`	`Split \| str \| None`	Keep only targets of tasks of this split, or `None` for all.	`None`

Returns:

Type	Description
`Table`	Targets in task ID and position order.

read_task_table ¶source

read_task_table(
    record_ids: Iterable[int] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> Table

Return the task table as Arrow, with object references as public IDs, without building Tasks.

The table has one row per task. Each row holds the type, prompt, rationale, the four scope columns, the configuration columns, ordered input_record_ids, ordered from_task_ids, and the metadata JSON string. Targets are in :meth:read_target_table.

Parameters:

Name	Type	Description	Default
`record_ids`	`Iterable[int] \| None`	Keep only tasks whose inputs include one of these Records, or `None` for all.	`None`
`split`	`Split \| str \| None`	Keep only tasks of this split, or `None` for all.	`None`

Returns:

Type	Description
`Table`	The tasks in task ID order.

read_tasks ¶source

read_tasks(
    records: Iterable[Record] | None = None,
    *,
    cached_records: Iterable[Record] = (),
    dataset_annotations: Mapping[int, Annotation]
    | None = None,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> tuple[Task, ...]

Build every selected task and restore all in-memory object references.

Parameters:

Name	Type	Description	Default
`records`	`Iterable[Record] \| None`	Input Records to select, or `None` for every task. Tasks reuse these objects.	`None`
`cached_records`	`Iterable[Record]`	Records already built, to reuse without restricting the selection.	`()`
`dataset_annotations`	`Mapping[int, Annotation] \| None`	The dataset's annotations, already built, by occurrence id, which tasks then reference instead of their own copies, or `None` to read them when a task needs them.	`None`
`split`	`Split \| str \| None`	Keep only tasks of this split, or `None` for every task.	`None`

Returns:

Type	Description
`tuple[Task, ...]`	Concrete tasks in stable ID order.

Raises:

Type	Description
`TimeFFormatError`	If a relationship refers to a missing object or has an invalid field.

read_tasks_by_id ¶source

read_tasks_by_id(
    task_ids: Collection[EntityId],
) -> dict[EntityId, Task]

Build the given tasks, sharing the Records they reference, a bounded batch at a time.

The tasks they derive from through from_tasks are built too.

Parameters:

Name	Type	Description	Default
`task_ids`	`Collection[EntityId]`	The task IDs, each once.	required

Returns:

Type	Description
`dict[EntityId, Task]`	The tasks by ID.

Raises:

Type	Description
`TimeFNotFoundError`	If an ID names no stored task.

record_clock_ids ¶source

record_clock_ids(
    record_ids: Collection[EntityId],
) -> dict[EntityId, int]

Return the clock of each of the given Records, building no object.

A clock ID names one clock within this catalog only. Records with one clock ID share one TimeOrigin when a read builds them.

Returns:

Type	Description
`dict[EntityId, int]`	The clock ID of each Record, by Record ID. An ID that names no Record is absent.

record_ids ¶source

record_ids() -> tuple[int, ...]

Return every stored Record ID in the order :meth:read_records yields them, without building a Record.

Returns:

Type	Description
`tuple[int, ...]`	The Record IDs in their stored read order.

run ¶source

run(query: Callable[[DuckDBPyConnection], _T]) -> _T

Run one read through the connection, which opens again first if it was closed.

Parameters:

Name	Type	Description	Default
`query`	`Callable[[DuckDBPyConnection], _T]`	Reads through the connection it is given.	required

Returns:

Type	Description
`_T`	What `query` returns.

Raises:

Type	Description
`TimeFReaderError`	If the connection closed while the read ran, through :meth:`close` or through the before-fork hook in another thread.
`TimeFFormatError`	If the catalog lacks a table, column, or type that the read expects.
`Error`	Any other DuckDB failure, such as I/O, memory, or an interrupt, unchanged.

signal_record_ids ¶source

signal_record_ids(
    signal_ids: Collection[EntityId],
) -> dict[EntityId, EntityId]

Return the ID of the Record that holds each of the given Signals, building no object.

Parameters:

Name	Type	Description	Default
`signal_ids`	`Collection[EntityId]`	The Signals' IDs.	required

Returns:

Type	Description
`dict[EntityId, EntityId]`	The Record ID of each Signal, by Signal ID.

Raises:

Type	Description
`TimeFNotFoundError`	If an ID names no stored Signal.

task_facts ¶source

task_facts() -> _TaskFacts

Return the per-database facts that task reads need, read once per reader.

Returns:

Type	Description
`_TaskFacts`	The IDs of tasks other tasks derive from, and whether any task carries annotations.

Raises:

Type	Description
`TimeFFormatError`	If two task rows share an ID.

control_schema ¶source

The single description of the control database's tables.

The SQL table definitions and the writer's Arrow batch schemas come from :data:CONTROL_TABLES. The audit also uses it to examine keys and references. Define schema changes there.

This is version 1 of the control schema. Every table except label_sets uses ids as keys. Records, Signals, and Tasks use the public ids that the caller sees. All other objects use the writer's storage ids.

Each label set has a unique name in a dataset. The label_sets table uses this name as its key. A foreign key stores the key of the row that it refers to.

Fixed-shape fields are typed columns, so SQL can filter them. Only the free-form metadata dictionaries are JSON. The database has no persisted constraints. The writer keeps :data:Table.unique and :data:Table.foreign_keys while it streams rows, and :func:timenet.format.control_audit.audit_control_database checks them again.

ANNOTATED_OBJECT_TYPES `module-attribute` ¶source

ANNOTATED_OBJECT_TYPES: frozenset[ObjectKind] = frozenset(
    ANNOTATED_OWNER_TABLES
)

The values annotation_occurrences.object_type can hold.

ANNOTATED_OWNER_TABLES `module-attribute` ¶source

ANNOTATED_OWNER_TABLES: dict[
    ObjectKind, tuple[str, str] | None
] = {
    ObjectKind.DATASET: None,
    ObjectKind.RECORD: ("records", "record_id"),
    ObjectKind.SOURCE: ("sources", "source_id"),
    ObjectKind.SIGNAL: ("signals", "signal_id"),
    ObjectKind.TASK: ("tasks", "task_id"),
}

The owner table and key column of each annotated object type. The Dataset has no owner row.

ANNOTATION_CONTENTS `module-attribute` ¶source

ANNOTATION_CONTENTS = Table(
    "annotation_contents",
    (
        _required("content_id", ColumnType.BIGINT),
        _required("name"),
        Column("value_kind", ColumnType.VARCHAR),
        Column("text_value", ColumnType.VARCHAR),
        Column("integer_value", ColumnType.BIGINT),
        Column("float_value", ColumnType.DOUBLE),
        Column("boolean_value", ColumnType.BOOLEAN),
        Column("text_list_value", ColumnType.VARCHAR_LIST),
        Column("unit", ColumnType.VARCHAR),
        Column("description", ColumnType.VARCHAR),
        _required("metadata", ColumnType.JSON),
    ),
    unique=(("content_id",),),
)

ANNOTATION_OCCURRENCES `module-attribute` ¶source

ANNOTATION_OCCURRENCES = Table(
    "annotation_occurrences",
    (
        _required("occurrence_id", ColumnType.BIGINT),
        _required("content_id", ColumnType.BIGINT),
        _required("object_type"),
        Column("object_id", ColumnType.BIGINT),
        _required("span_type"),
        Column("start_us", ColumnType.BIGINT),
        Column("end_us", ColumnType.BIGINT),
        Column("provenance", ColumnType.JSON),
        Column("confidence", ColumnType.DOUBLE),
        _required("metadata", ColumnType.JSON),
    ),
    unique=(("occurrence_id",),),
    foreign_keys=(
        ForeignKey(
            "content_id",
            "annotation_contents",
            "content_id",
        ),
    ),
)

The owner is (object_type, object_id). object_id is the public id of a Record, Signal, or Task, or the storage id of a Source. Ids are unique only within one kind, so the type is part of the reference. object_id is NULL exactly when the owner is the Dataset.

AXES `module-attribute` ¶source

AXES = Table(
    "axes",
    (
        _required("axis_id", ColumnType.BIGINT),
        _required("axis_type"),
        Column("period_numerator_us", ColumnType.BIGINT),
        Column("period_denominator", ColumnType.BIGINT),
        Column("start_index", ColumnType.BIGINT),
        Column("offset_us", ColumnType.BIGINT),
        Column("first_us", ColumnType.BIGINT),
        Column("last_us", ColumnType.BIGINT),
    ),
    unique=(("axis_id",),),
)

CLOCKS `module-attribute` ¶source

CLOCKS = Table(
    "clocks",
    (
        _key("clock_id"),
        Column("start_time_us", ColumnType.BIGINT),
    ),
    unique=(("clock_id",),),
)

CONTROL_METADATA `module-attribute` ¶source

CONTROL_METADATA = Table(
    "control_metadata",
    (_required("key"), _required("value")),
    unique=(("key",),),
)

CONTROL_TABLES `module-attribute` ¶source

CONTROL_TABLES: tuple[Table, ...] = (
    CONTROL_METADATA,
    IMPORTS,
    EXTERNAL_VALUES,
    IMPORTED_OBJECTS,
    DATASETS,
    RECORDS,
    CLOCKS,
    SOURCES,
    AXES,
    SIGNALS,
    SIGNAL_CHUNKS,
    ANNOTATION_CONTENTS,
    ANNOTATION_OCCURRENCES,
    LABEL_SETS,
    TASKS,
    TASK_TARGETS,
    TASK_RECORD_REFS,
    TASK_ANNOTATION_REFS,
    TASK_DEPENDENCIES,
)

Every control table, in creation order.

DATASETS `module-attribute` ¶source

DATASETS = Table(
    "datasets",
    (_required("dataset_id"),),
    unique=(("dataset_id",),),
)

Exactly one row, the dataset the file describes.

EXTERNAL_OWNER_TABLES `module-attribute` ¶source

EXTERNAL_OWNER_TABLES: dict[ObjectKind, tuple[str, str]] = {
    ObjectKind.TASK: ("tasks", "task_id"),
    ObjectKind.ANNOTATION_CONTENT: (
        "annotation_contents",
        "content_id",
    ),
    ObjectKind.AXIS: ("axes", "axis_id"),
}

The owner table and key column of each object kind whose values can be stored externally.

EXTERNAL_VALUES `module-attribute` ¶source

EXTERNAL_VALUES = Table(
    "external_values",
    (
        _required("object_kind"),
        _required("object_id", ColumnType.BIGINT),
        _required("field"),
        _key("position"),
        Column("source_key", ColumnType.VARCHAR),
        _required("value_path"),
        _key("row_index"),
        _required("value_kind"),
        _key("n_values"),
        Column("value_digest", ColumnType.VARCHAR),
    ),
    unique=(
        ("object_kind", "object_id", "field", "position"),
    ),
)

The owner is (object_kind, object_id): a Task by its public id, or an annotation content row or an irregular axis by the storage id that this write gave its row. Ids are unique only within one kind, so the kind is part of the reference. Signals that share an axis share its row, so the axis's time offsets are stored once.

IMPORTED_OBJECTS `module-attribute` ¶source

IMPORTED_OBJECTS = Table(
    "imported_objects",
    (
        _required("alias"),
        _required("object_kind"),
        _required("parent_id", ColumnType.BIGINT),
        _required("object_id", ColumnType.BIGINT),
    ),
    unique=(("alias", "object_kind", "parent_id"),),
    foreign_keys=(ForeignKey("alias", "imports", "alias"),),
)

Local object IDs indexed by parent alias, object kind, and original parent ID.

IMPORTS `module-attribute` ¶source

IMPORTS = Table(
    "imports",
    (_required("alias"), _required("parent_ref")),
    unique=(("alias",), ("parent_ref",)),
)

Catalog lookup projection of the aliases and exact releases in Manifest.parent_cards.

LABEL_SETS `module-attribute` ¶source

LABEL_SETS = Table(
    "label_sets",
    (
        _required("name"),
        _required("labels", ColumnType.LABEL_LIST),
        _required("multi_label", ColumnType.BOOLEAN),
    ),
    unique=(("name",),),
)

Each row stores one label set that a classification task uses.

The labels column holds one {name, description} entry per label, in label order. If a label has no description, description is NULL. Tasks refer to a set by its name in tasks.label_set.

LAST_ID_PREFIX `module-attribute` ¶source

LAST_ID_PREFIX = 'last_id.'

Prefix of the control_metadata keys that store the last id used per entity kind.

RECORDS `module-attribute` ¶source

RECORDS = Table(
    "records",
    (
        _required("record_id", ColumnType.BIGINT),
        _key("clock_id"),
        Column("time_span_start_us", ColumnType.BIGINT),
        Column("time_span_end_us", ColumnType.BIGINT),
        _required("metadata", ColumnType.JSON),
        _required("read_order", ColumnType.BIGINT),
    ),
    unique=(("record_id",), ("read_order",)),
    foreign_keys=(
        ForeignKey("clock_id", "clocks", "clock_id"),
    ),
)

SIGNALS `module-attribute` ¶source

SIGNALS = Table(
    "signals",
    (
        _required("signal_id", ColumnType.BIGINT),
        _key("source_id"),
        _required("name"),
        _key("axis_id"),
        _required("spec_type"),
        _required("spec_name"),
        Column("unit", ColumnType.VARCHAR),
        _required("dtype"),
        _required("categories", ColumnType.VARCHAR_LIST),
        _required("value_shape", ColumnType.BIGINT_LIST),
        _required(
            "dimension_names", ColumnType.VARCHAR_LIST
        ),
        _required("nullable", ColumnType.BOOLEAN),
        _key("n_values"),
        _required("metadata", ColumnType.JSON),
        _required("modality"),
    ),
    unique=(("signal_id",),),
    foreign_keys=(
        ForeignKey("source_id", "sources", "source_id"),
        ForeignKey("axis_id", "axes", "axis_id"),
    ),
)

SIGNAL_CHUNKS `module-attribute` ¶source

SIGNAL_CHUNKS = Table(
    "signal_chunks",
    (
        _key("signal_id"),
        _key("chunk_index"),
        _required("value_path"),
        Column("source_key", ColumnType.VARCHAR),
        _key("chunk_major_index"),
        Column("chunk_minor_index", ColumnType.BIGINT),
        _key("n_values"),
    ),
    unique=(("signal_id", "chunk_index"),),
    foreign_keys=(
        ForeignKey("signal_id", "signals", "signal_id"),
    ),
)

SOURCES `module-attribute` ¶source

SOURCES = Table(
    "sources",
    (
        _required("source_id", ColumnType.BIGINT),
        _key("record_id"),
        Column("parent_source_id", ColumnType.BIGINT),
        _required("name"),
        _required("metadata", ColumnType.JSON),
    ),
    unique=(("source_id",),),
    foreign_keys=(
        ForeignKey("record_id", "records", "record_id"),
        ForeignKey(
            "parent_source_id", "sources", "source_id"
        ),
    ),
)

TABLES `module-attribute` ¶source

TABLES: dict[str, Table] = {
    (table.name): table for table in CONTROL_TABLES
}

The control tables by name.

TASKS `module-attribute` ¶source

TASKS = Table(
    "tasks",
    (
        _required("task_id", ColumnType.BIGINT),
        _required("task_type"),
        Column("prompt", ColumnType.VARCHAR),
        _required(
            "input_modalities", ColumnType.VARCHAR_LIST
        ),
        Column("split", ColumnType.VARCHAR),
        Column("scope_type", ColumnType.VARCHAR),
        Column("scope_start", ColumnType.BIGINT),
        Column("scope_end", ColumnType.BIGINT),
        Column("scope_signal_ids", ColumnType.BIGINT_LIST),
        _required("has_inline_targets", ColumnType.BOOLEAN),
        Column("label_set", ColumnType.VARCHAR),
        Column("unit", ColumnType.VARCHAR),
        Column("target_name", ColumnType.VARCHAR),
        Column("mode", ColumnType.VARCHAR),
        Column("rationale", ColumnType.VARCHAR),
        _required("metadata", ColumnType.JSON),
    ),
    unique=(("task_id",),),
    foreign_keys=(
        ForeignKey(
            "scope_signal_ids",
            "signals",
            "signal_id",
            many=True,
        ),
        ForeignKey("label_set", "label_sets", "name"),
    ),
)

TASK_ANNOTATION_REFS `module-attribute` ¶source

TASK_ANNOTATION_REFS = Table(
    "task_annotation_refs",
    (
        _key("task_id"),
        _required("field"),
        _key("position"),
        _key("occurrence_id"),
    ),
    unique=(("task_id", "field", "position"),),
    foreign_keys=(
        ForeignKey("task_id", "tasks", "task_id"),
        ForeignKey(
            "occurrence_id",
            "annotation_occurrences",
            "occurrence_id",
        ),
    ),
)

TASK_DEPENDENCIES `module-attribute` ¶source

TASK_DEPENDENCIES = Table(
    "task_dependencies",
    (
        _key("task_id"),
        _key("position"),
        _key("parent_task_id"),
    ),
    unique=(("task_id", "position"),),
    foreign_keys=(
        ForeignKey("task_id", "tasks", "task_id"),
        ForeignKey("parent_task_id", "tasks", "task_id"),
    ),
)

TASK_RECORD_REFS `module-attribute` ¶source

TASK_RECORD_REFS = Table(
    "task_record_refs",
    (
        _key("task_id"),
        _required("field"),
        _key("position"),
        _key("record_id"),
    ),
    unique=(("task_id", "field", "position"),),
    foreign_keys=(
        ForeignKey("task_id", "tasks", "task_id"),
        ForeignKey("record_id", "records", "record_id"),
    ),
)

TASK_TARGETS `module-attribute` ¶source

TASK_TARGETS = Table(
    "task_targets",
    (
        _key("task_id"),
        _key("position"),
        _required("target_kind"),
        Column("text_value", ColumnType.VARCHAR),
        Column("integer_value", ColumnType.BIGINT),
        Column("float_value", ColumnType.DOUBLE),
        Column("boolean_value", ColumnType.BOOLEAN),
        Column("record_id", ColumnType.BIGINT),
        Column("signal_id", ColumnType.BIGINT),
        Column("span_start", ColumnType.BIGINT),
        Column("span_end", ColumnType.BIGINT),
        Column("span_signal_ids", ColumnType.BIGINT_LIST),
    ),
    unique=(("task_id", "position"),),
    foreign_keys=(
        ForeignKey("task_id", "tasks", "task_id"),
        ForeignKey("record_id", "records", "record_id"),
        ForeignKey("signal_id", "signals", "signal_id"),
        ForeignKey(
            "span_signal_ids",
            "signals",
            "signal_id",
            many=True,
        ),
    ),
)

The value columns match :class:timenet.format.task_target_row.TargetValueRow. A span names its series in span_signal_ids: one Signal for a step span, and any number for a time span.

Column `dataclass` ¶source

One column of a control table.

name `instance-attribute` ¶source

name: str

nullable `class-attribute` `instance-attribute` ¶source

nullable: bool = True

type `instance-attribute` ¶source

type: ColumnType

ddl ¶source

ddl() -> str

Return the column's CREATE TABLE fragment.

Returns:

Type	Description
`str`	The column definition without a trailing comma.

ColumnType ¶source

Bases: Enum

The DuckDB column types the control schema uses, each with the Arrow type that feeds it.

The writer builds every batch with the table's Arrow schema, so a value of the wrong Python type fails in Arrow, naming the column, instead of being cast by DuckDB.

BIGINT `class-attribute` `instance-attribute` ¶source

BIGINT = ('BIGINT', pa.int64())

BIGINT_LIST `class-attribute` `instance-attribute` ¶source

BIGINT_LIST = ('BIGINT[]', pa.list_(pa.int64()))

BOOLEAN `class-attribute` `instance-attribute` ¶source

BOOLEAN = ('BOOLEAN', pa.bool_())

DOUBLE `class-attribute` `instance-attribute` ¶source

DOUBLE = ('DOUBLE', pa.float64())

JSON `class-attribute` `instance-attribute` ¶source

JSON = ('JSON', pa.string())

LABEL_LIST `class-attribute` `instance-attribute` ¶source

LABEL_LIST = (
    "STRUCT(name VARCHAR, description VARCHAR)[]",
    pa.list_(
        pa.struct(
            [
                ("name", pa.string()),
                ("description", pa.string()),
            ]
        )
    ),
)

VARCHAR `class-attribute` `instance-attribute` ¶source

VARCHAR = ('VARCHAR', pa.string())

VARCHAR_LIST `class-attribute` `instance-attribute` ¶source

VARCHAR_LIST = ('VARCHAR[]', pa.list_(pa.string()))

arrow `instance-attribute` ¶source

arrow = arrow

The Arrow type of the column in a writer batch.

sql `instance-attribute` ¶source

sql = sql

The type name in CREATE TABLE.

ForeignKey `dataclass` ¶source

A column whose non-NULL values must exist in another table's column.

column `instance-attribute` ¶source

column: str

many `class-attribute` `instance-attribute` ¶source

many: bool = False

Whether column is a list column whose every element must exist.

references `instance-attribute` ¶source

references: str

table `instance-attribute` ¶source

table: str

Table `dataclass` ¶source

One control table: its columns in storage order, and the invariants it holds.

arrow_schema `property` ¶source

arrow_schema: Schema

The schema of a writer batch for this table, one Arrow field per column.

column_names `property` ¶source

column_names: tuple[str, ...]

The column names in storage order.

columns `instance-attribute` ¶source

columns: tuple[Column, ...]

foreign_keys `class-attribute` `instance-attribute` ¶source

foreign_keys: tuple[ForeignKey, ...] = ()

name `instance-attribute` ¶source

name: str

unique `class-attribute` `instance-attribute` ¶source

unique: tuple[tuple[str, ...], ...] = ()

Column sets that identify one row.

ddl ¶source

ddl() -> str

Return the table's CREATE TABLE statement.

Returns:

Type	Description
`str`	The statement, with a trailing semicolon.

schema_ddl ¶source

schema_ddl() -> str

Return the statements that create every table of the control schema.

Returns:

Type	Description
`str`	DDL for :func:`timenet.format.duckdb.create_control_schema`.

control_writer ¶source

Write the in-memory TimeF hierarchy into control.duckdb.

DuckDBControlWriter ¶source

Serialize the relational part of one immutable TimeF version.

external `instance-attribute` ¶source

external = external

path `instance-attribute` ¶source

path = Path(path)

write_hierarchy ¶source

write_hierarchy(
    dataset: TimeFDataset,
    placements: dict[tuple[int, int], ChunkPlacement]
    | None = None,
    *,
    tasks: Iterable[Task] | None = None,
) -> dict[str, int]

Write records, sources, signals, axes, and their annotations atomically.

Parameters:

Name	Type	Description	Default
`dataset`	`TimeFDataset`	The complete in-memory hierarchy.	required
`placements`	`dict[tuple[int, int], ChunkPlacement] \| None`	Values-plane chunks keyed by `(signal_id, chunk_index)`.	`None`
`tasks`	`Iterable[Task] \| None`	A previously validated task stream, or `None` to use the dataset's tasks.	`None`

Returns:

Type	Description
`dict[str, int]`	Counts keyed by concrete task type.

Raises:

Type	Description
`TimeFValidationError`	If the hierarchy refers to an object outside the dataset, repeats a task id, holds a value that JSON cannot store, or declares Signal lengths that do not match its axis offsets or value chunks.

duckdb ¶source

DuckDB schema and connection helpers for the TimeF relational control plane.

CATALOG_ERRORS `module-attribute` ¶source

CATALOG_ERRORS = (
    duckdb.CatalogException,
    duckdb.BinderException,
    duckdb.ConversionException,
)

DuckDB errors that mean a catalog lacks a table, column, or type that a read expects.

CONTROL_FILE `module-attribute` ¶source

CONTROL_FILE = 'control.duckdb'

Name of the relational control-plane database in a TimeF version.

CONTROL_SCHEMA_VERSION `module-attribute` ¶source

CONTROL_SCHEMA_VERSION = 1

Schema version written into :data:CONTROL_FILE.

check_control_schema ¶source

check_control_schema(connection: DuckDBPyConnection) -> int

Return the version of a supported control database.

Parameters:

Name	Type	Description	Default
`connection`	`DuckDBPyConnection`	An open TimeF control database.	required

Raises:

Type	Description
`TimeFFormatError`	If the schema metadata is missing, malformed, or unsupported.

connect_control ¶source

connect_control(
    path: Path, *, read_only: bool = False
) -> DuckDBPyConnection

Open a TimeF control database.

Parameters:

Name	Type	Description	Default
`path`	`Path`	Local path to `control.duckdb`.	required
`read_only`	`bool`	Open an immutable published database without write access.	`False`

Returns:

Type	Description
`DuckDBPyConnection`	An open DuckDB connection owned by the caller.

create_control_schema ¶source

create_control_schema(
    connection: DuckDBPyConnection,
) -> None

Create every control-plane table in one transaction.

Parameters:

Name	Type	Description	Default
`connection`	`DuckDBPyConnection`	A writable connection to a new database.	required

transaction ¶source

transaction(
    connection: DuckDBPyConnection,
) -> Generator[None]

Commit a block atomically or roll it back when any operation fails.

Parameters:

Name	Type	Description	Default
`connection`	`DuckDBPyConnection`	The connection that owns the transaction.	required

Yields:

Type	Description
`Generator[None]`	Control while the transaction is open.

schemas ¶source

Pinned Arrow schema of the Parquet value shards.

Shards hold no public ids. The DuckDB control plane maps each signal to its chunk placements.

shard_schema ¶source

shard_schema(
    value_type: DataType = _DEFAULT_VALUES_TYPE,
) -> Schema

Return the waveform shard schema.

Parameters:

Name	Type	Description	Default
`value_type`	`DataType`	The element type of the `values` list (defaults to `float32`).	`_DEFAULT_VALUES_TYPE`

Returns:

Type	Description
`Schema`	The Arrow schema.

storage ¶source

Write-time collection of the payload sources a new release reads, pinned to their files.

StorageSources ¶source

Collect the releases that store a new release's reused payloads, and pin the files it reads.

With a target registry, each payload source is checked there at its first reuse. The registry must hold the source with the same records of the reused files as the release the values were read from, and the source must not read payloads of the new release itself.

registry `instance-attribute` ¶source

registry = registry

sources `property` ¶source

sources: dict[SourceKey, PayloadSource]

Each payload source with the files the new release reads from it, for the manifest.

target `instance-attribute` ¶source

target = target

add ¶source

add(
    release: DatasetVersion,
    location: PayloadLocation,
    *,
    nullable: bool = False,
) -> SourceKey

Check a reused payload's files and pin them on the release that stores them.

Parameters:

Name	Type	Description	Default
`release`	`DatasetVersion`	The release whose catalog records `location`. The location's source names one of its payload sources, or is `None` when `release` stores the payload itself.	required
`location`	`PayloadLocation`	Where the payload's bytes live.	required
`nullable`	`bool`	Whether the spec of a signal array is nullable. Zarr stores the array's validity in parallel arrays, which the source must then list too.	`False`

Returns:

Type	Description
`SourceKey`	Its source key in the new release: the exact release as `org/name@version`.

Raises:

Type	Description
`TimeFValidationError`	If the storing release is the output release, its manifest lacks a file of the payload, or the target registry holds a source that reads payloads of the output release.
`TimeNetDatasetNotFoundError`	If the target registry does not hold the source.
`TimeFFormatError`	If the target registry's source records a reused file differently.

stored_array ¶source

Signal arrays stored as chunks in the Parquet or Zarr values files of a release.

ChunkLocation `dataclass` ¶source

Locate a signal array stored as chunks in the values files of one release.

The catalog records each chunk's file, coordinates, and length, so the array's place in storage is known without opening a values file.

chunks `instance-attribute` ¶source

chunks: tuple[StoredChunk, ...]

The array's chunks in step order.

source `class-attribute` `instance-attribute` ¶source

source: SourceKey | None = None

The payload source that stores the chunks, or None for the release whose catalog records them.

ChunkReader ¶source

Bases: Protocol

Find and read the stored chunks of one release's signal arrays.

A release's TimeFReader is one: it finds the chunks of a batch of signals at a time, and reads them from whichever release stores them.

StoredArray `dataclass` ¶source

Bases: ArrayHandle

Refer to one signal's array, stored as chunks in the values files of a release.

The handle holds the release's reader and the signal's ID, spec, and number of values. A loaded handle also keeps the array and the location of its chunks in a write-once cell, so it reads, slices, and locates them without storage access. An unloaded handle holds no values: every read goes to the chunks, and read_steps reads only the chunks that cover the range. Handles from one reader for one signal are equal, whether loaded or not.

is_loaded `property` ¶source

is_loaded: bool

Whether the handle keeps the values, so reads need no storage access.

n_values `instance-attribute` ¶source

n_values: int

Number of values that the signal declares, which a read of every chunk must return.

reader `class-attribute` `instance-attribute` ¶source

reader: ChunkReader = dataclass_field(repr=False)

Reader of the release whose catalog records the signal.

signal_id `instance-attribute` ¶source

signal_id: int

ID of the signal in that release's catalog.

spec `instance-attribute` ¶source

spec: TimeSeriesSpec

Spec that the stored values decode to.

read ¶source

read() -> Array

Return the kept values, or the values of every chunk.

Returns:

Type	Description
`Array`	The signal's values.

Raises:

Type	Description
`TimeFFormatError`	If the chunks hold another number of values than the signal declares.

read_numpy ¶source

read_numpy() -> ndarray

Return the values as a read-only NumPy array, shaped (n_steps, *value_shape).

The default converts :meth:read. A handle whose data is NumPy already returns it without building Arrow.

Returns:

Type	Description
`ndarray`	The values, in the NumPy dtype of their Arrow type.

Raises:

Type	Description
`TimeFNullValuesError`	If the values contain nulls, which NumPy cannot hold. Read them with :meth:`read` instead.

read_steps ¶source

read_steps(start: int, stop: int) -> Array

Return the half-open step range [start, stop).

A loaded handle slices the values it keeps. An unloaded one reads only the chunks that cover the range.

Returns:

Type	Description
`Array`	The steps' values, in the form :meth:`read` returns.

stored_values ¶source

Read and write text, string lists, and time offsets stored in Parquet payloads.

ANNOTATION_COLUMNS `module-attribute` ¶source

ANNOTATION_COLUMNS = StoredValueColumns(
    kind=ObjectKind.ANNOTATION_CONTENT,
    identity="content_id",
    fields={
        PackedField.VALUE: {
            PackedValueKind.TEXT: "text_value",
            PackedValueKind.TEXT_LIST: "text_list_value",
        }
    },
)

Packable columns of the annotation table.

INLINE_LIMIT `module-attribute` ¶source

INLINE_LIMIT = 4096

The largest JSON-encoded text size, in bytes, that the writer keeps in the catalog.

The writer stores larger text and lazy text handles in separate payload files to keep the catalog small.

LOCATOR_TABLE `module-attribute` ¶source

LOCATOR_TABLE = ModelTable[PackedValueLocator](
    EXTERNAL_VALUES
)

PayloadLocation `module-attribute` ¶source

PayloadLocation = PackedLocation | ChunkLocation

Where a stored value's bytes live: packed text or time offsets, or a signal array's chunks.

TARGET_COLUMNS `module-attribute` ¶source

TARGET_COLUMNS = StoredValueColumns(
    kind=ObjectKind.TASK,
    identity="task_id",
    fields={
        PackedField.TARGETS: {
            PackedValueKind.TEXT: "text_value"
        }
    },
    position="position",
)

Packable columns of the task-target table.

TASK_COLUMNS `module-attribute` ¶source

TASK_COLUMNS = StoredValueColumns(
    kind=ObjectKind.TASK,
    identity="task_id",
    fields={
        PackedField.PROMPT: {
            PackedValueKind.TEXT: "prompt"
        },
        PackedField.RATIONALE: {
            PackedValueKind.TEXT: "rationale"
        },
    },
)

Packable columns of the task table.

CatalogValue `dataclass` ¶source

Hold a value kept in its catalog row, including a legitimate absent value.

value `instance-attribute` ¶source

value: object

ExternalObjectFieldKey `dataclass` ¶source

Identify an external field on an object, including its position in an ordered field.

The object is named the way the catalog names it: a task by its public id, an annotation's content by the storage id of its annotation_contents row, and an irregular axis by the storage id of its axes row.

field `instance-attribute` ¶source

field: PackedField

object_id `instance-attribute` ¶source

object_id: int

object_kind `instance-attribute` ¶source

object_kind: ObjectKind

position `class-attribute` `instance-attribute` ¶source

position: int = 0

ExternalValue `dataclass` ¶source

Identify a value written to a payload instead of the catalog.

locator `instance-attribute` ¶source

locator: PackedValueLocator

PackedContentKey `dataclass` ¶source

Name one packed value by its payload file's checksum and its place in the file, without a read.

The checksum identifies the file's bytes, so equal keys mean equal values, and readers of one release, or of releases that share a payload file, give the same key. The key compares equal only to another PackedContentKey, so it never equals the key of text in memory, which is a string or a tuple of strings.

checksum `instance-attribute` ¶source

checksum: str

The payload file's checksum, as the manifest records it.

kind `instance-attribute` ¶source

kind: PackedValueKind

Kind of the value, which names the payload column that holds it.

row `instance-attribute` ¶source

row: int

Row of the value in the payload file.

PackedField ¶source

Bases: StrEnum

Name the object fields that can store external values.

PROMPT `class-attribute` `instance-attribute` ¶source

PROMPT = 'prompt'

RATIONALE `class-attribute` `instance-attribute` ¶source

RATIONALE = 'rationale'

TARGETS `class-attribute` `instance-attribute` ¶source

TARGETS = 'targets'

TIME_OFFSETS `class-attribute` `instance-attribute` ¶source

TIME_OFFSETS = 'time_offsets'

VALUE `class-attribute` `instance-attribute` ¶source

VALUE = 'value'

PackedLocation `dataclass` ¶source

Locate one value packed in a Parquet payload file of a release.

The catalog records every field, so the value's release, kind, length, and digest are known without opening the payload file.

digest `class-attribute` `instance-attribute` ¶source

digest: bytes | None = None

SHA-256 digest of time offsets, which identifies them without a read. None for text.

kind `instance-attribute` ¶source

kind: PackedValueKind

Kind of the value, which names the payload column that holds it.

length `instance-attribute` ¶source

length: int

Number of code points in a string, of items in a string list, or of time offsets.

path `instance-attribute` ¶source

path: str

Payload file, relative to the root of the release that stores it.

row `instance-attribute` ¶source

row: int

Row of the value in the payload file.

source `class-attribute` `instance-attribute` ¶source

source: SourceKey | None = None

The payload source that stores the file, or None for the release whose catalog records it.

PackedValueKind ¶source

Bases: StrEnum

Represent the value shape stored at a packed payload location.

TEXT `class-attribute` `instance-attribute` ¶source

TEXT = 'text'

TEXT_LIST `class-attribute` `instance-attribute` ¶source

TEXT_LIST = 'texts'

TIME_OFFSETS `class-attribute` `instance-attribute` ¶source

TIME_OFFSETS = 'time_offsets'

PackedValueLocator ¶source

Bases: TimeFModel

Validate the catalog location and value kind of one external field.

field `instance-attribute` ¶source

field: PackedField

key `property` ¶source

key: ExternalObjectFieldKey

The validated object field addressed by this locator.

location `property` ¶source

location: PackedLocation

Where the value is packed.

model_config `class-attribute` `instance-attribute` ¶source

model_config = ConfigDict(
    extra="forbid",
    frozen=True,
    revalidate_instances="always",
    serialize_by_alias=True,
    validate_by_name=True,
    validate_default=True,
)

n_values `class-attribute` `instance-attribute` ¶source

n_values: int = Field(strict=True, ge=0)

object_id `class-attribute` `instance-attribute` ¶source

object_id: int = Field(strict=True, gt=0)

object_kind `instance-attribute` ¶source

object_kind: ObjectKind

position `class-attribute` `instance-attribute` ¶source

position: int = Field(strict=True, ge=0)

row_index `class-attribute` `instance-attribute` ¶source

row_index: int = Field(strict=True, ge=0)

source_key `class-attribute` `instance-attribute` ¶source

source_key: SourceKey | None = Field(
    default=None, strict=True, min_length=1
)

value_digest `class-attribute` `instance-attribute` ¶source

value_digest: str | None = Field(
    default=None,
    pattern=f"^{CHECKSUM_PREFIX}[0-9a-f]{64}$",
    strict=True,
)

value_kind `instance-attribute` ¶source

value_kind: PackedValueKind

value_path `class-attribute` `instance-attribute` ¶source

value_path: str = Field(strict=True, min_length=1)

PackedValueReader ¶source

Read one release's packed values through a bounded cache of decoded payload files.

A value whose location names a source is read from that payload source of the release.

version `instance-attribute` ¶source

version = version

checksum ¶source

checksum(location: PackedLocation) -> str

Return the checksum that the release storing the location's payload file records for it.

The location's source names that release: one of this release's payload sources, or this release itself for None. Its manifest lists the file, so the checksum needs no read.

Returns:

Type	Description
`str`	The file's checksum, with its `sha256:` prefix.

Raises:

Type	Description
`TimeFFormatError`	If the source is unknown, or its manifest does not list the file.

close ¶source

close() -> None

Release cached payload tables.

content_key ¶source

content_key(location: PackedLocation) -> PackedContentKey

Return the key that names a packed value by its payload file's checksum, without a read.

Returns:

Type	Description
`PackedContentKey`	The checksum the manifest records for the location's file, with the row and kind.

Raises:

Type	Description
`TimeFFormatError`	If the manifest does not list the location's payload file.

handle ¶source

handle(
    location: PackedLocation,
) -> StoredText | StoredTextList

Return a text handle of the location's value kind, without reading the payload.

Returns:

Type	Description
`StoredText \| StoredTextList`	A `StoredText` for a string, or a `StoredTextList` for a list of strings.

Raises:

Type	Description
`TimeFFormatError`	If the manifest does not list the location's payload file.
`TimeFValidationError`	If the location holds time offsets, whose `StoredOffsets` also needs their axis.

handles ¶source

handles(
    connection: DuckDBPyConnection,
    kind: ObjectKind,
    object_ids: Iterable[int],
) -> dict[
    ExternalObjectFieldKey, StoredText | StoredTextList
]

Return text handles for one batch of objects' packed values without opening payload files.

Returns:

Type	Description
`dict[ExternalObjectFieldKey, StoredText \| StoredTextList]`	Handles keyed by object kind, object id, field, and position.

locations `staticmethod` ¶source

locations(
    connection: DuckDBPyConnection,
    kind: ObjectKind,
    object_ids: Iterable[int],
) -> dict[ExternalObjectFieldKey, PackedLocation]

Return where one batch of objects' values are packed, without opening payload files.

Parameters:

Name	Type	Description	Default
`connection`	`DuckDBPyConnection`	The release's open control database.	required
`kind`	`ObjectKind`	The kind of the objects that own the values.	required
`object_ids`	`Iterable[int]`	The catalog ids of the objects, such as task ids or annotation content ids.	required

Returns:

Type	Description
`dict[ExternalObjectFieldKey, PackedLocation]`	Locations keyed by object kind, object id, field, and position.

Raises:

Type	Description
`TimeFFormatError`	If the catalog has no locator table, or a locator is invalid or duplicated.

offsets ¶source

offsets(
    location: PackedLocation,
    *,
    axis_id: int,
    first: int,
    last: int,
) -> StoredOffsets

Return a handle to one irregular axis's packed time offsets, without reading the payload.

Parameters:

Name	Type	Description	Default
`location`	`PackedLocation`	Where the time offsets are packed.	required
`axis_id`	`int`	The axis's catalog ID, for error messages.	required
`first`	`int`	The first time offset the axis declares, in microseconds.	required
`last`	`int`	The last time offset the axis declares, in microseconds.	required

Returns:

Type	Description
`StoredOffsets`	A `StoredOffsets` that checks the time offsets against the axis on its first read.

Raises:

Type	Description
`TimeFFormatError`	If the manifest does not list the location's payload file.

read_scalar ¶source

read_scalar(location: PackedLocation) -> Scalar

Read a packed value from the payload file of the release that stores it.

Returns:

Type	Description
`Scalar`	The Arrow scalar at the location's file and row: a string, or a list of strings or time offsets.

Raises:

Type	Description
`TimeFFormatError`	If the payload file's schema, the location, or the stored value is invalid.

PackedValueWriter ¶source

Pack new values into Parquet payload files, and reuse the payloads that store read values.

bytes `instance-attribute` ¶source

bytes = 0

files `instance-attribute` ¶source

files: list[str] = []

locators `instance-attribute` ¶source

locators: list[PackedValueLocator] = []

root `instance-attribute` ¶source

root = root

sources `instance-attribute` ¶source

sources = sources

text_lists `instance-attribute` ¶source

text_lists: list[list[str] | None] = []

texts `instance-attribute` ¶source

texts: list[str | None] = []

time_offsets `instance-attribute` ¶source

time_offsets: list[Array | None] = []

finish ¶source

finish(connection: DuckDBPyConnection) -> None

Flush payload files and store the remaining catalog locators.

flush ¶source

flush() -> None

Write one packed group and release its buffered values.

pack ¶source

pack(
    kind: ObjectKind,
    object_id: int,
    field: PackedField,
    value: object,
    position: int = 0,
    *,
    digest: bytes | None = None,
) -> CatalogValue | ExternalValue

Store a value in the catalog or a separate payload file.

Time offsets, :class:LazyText, and :class:LazyTextList go into payload files. If other text exceeds :data:INLINE_LIMIT, it also goes into a payload file. Stored handles reuse their existing payload rows.

Parameters:

Name	Type	Description	Default
`kind`	`ObjectKind`	The kind of object that owns the value.	required
`object_id`	`int`	The object id in the output catalog: a task id, an annotation content id, or an axis id.	required
`field`	`PackedField`	The field that holds the value.	required
`value`	`object`	The text, time offsets, or other catalog value to store.	required
`position`	`int`	The position in a field with several values, such as task targets.	`0`
`digest`	`bytes \| None`	An optional `offsets_digest()` result for time offsets. If omitted, this method calculates it.	`None`

Returns:

Type	Description
`CatalogValue \| ExternalValue`	A catalog value or a locator that identifies the value in a payload file.

Raises:

Type	Description
`TimeFValidationError`	If the locator is invalid for this field, the output release stores a reused payload, or the source release lacks a reused payload.

write_locators ¶source

write_locators(
    connection: DuckDBPyConnection, *, at_least: int = 0
) -> None

Store the locators packed so far in the catalog and release them.

The catalog writer calls this once per batch inside its transaction, so a streamed source never holds all of its locators.

Parameters:

Name	Type	Description	Default
`connection`	`DuckDBPyConnection`	The open write transaction.	required
`at_least`	`int`	Keep the locators while there are fewer than this many.	`0`

StoredOffsets `dataclass` ¶source

Bases: ArrayHandle

Refer to the time offsets of one irregular axis, packed in a Parquet payload.

The handle holds the reader of the release's payloads, the time offsets' location, and the axis's ID and endpoints. Its first read() checks the time offsets against the location and the axis and keeps a compact copy in a write-once cell, so from then on the handle is loaded and reads need no payload access. The location records the number of time offsets and their digest, so both are known without a read.

axis_id `instance-attribute` ¶source

axis_id: int

Catalog ID of the axis the time offsets belong to.

digest `property` ¶source

digest: bytes

The time offsets' SHA-256 digest, recorded in their location.

The catalog stores a digest with every axis's time offsets, so only a location built by hand can lack one.

Raises:

Type	Description
`TimeFFormatError`	If the location records no digest.

first `instance-attribute` ¶source

first: int

First time offset the axis declares, in microseconds.

is_loaded `property` ¶source

is_loaded: bool

Whether a read has kept the time offsets.

last `instance-attribute` ¶source

last: int

Last time offset the axis declares, in microseconds.

length `property` ¶source

length: int

The number of time offsets, recorded in their location.

store `class-attribute` `instance-attribute` ¶source

store: PackedValueReader = dataclass_field(repr=False)

read ¶source

read() -> Array

Return the time offsets, read and checked on the first call and kept.

Returns:

Type	Description
`Array`	One int64 microsecond offset per step.

read_numpy ¶source

read_numpy() -> ndarray

Return the values as a read-only NumPy array, shaped (n_steps, *value_shape).

The default converts :meth:read. A handle whose data is NumPy already returns it without building Arrow.

Returns:

Type	Description
`ndarray`	The values, in the NumPy dtype of their Arrow type.

Raises:

Type	Description
`TimeFNullValuesError`	If the values contain nulls, which NumPy cannot hold. Read them with :meth:`read` instead.

read_steps ¶source

read_steps(start: int, stop: int) -> Array

Return the half-open step range [start, stop).

The default slices :meth:read, so a stop past the end reads to the end. A stored handle reads only the chunks that cover the range.

Parameters:

Name	Type	Description	Default
`start`	`int`	First step, inclusive, and not negative.	required
`stop`	`int`	Last step, exclusive, and not before `start`.	required

Returns:

Type	Description
`Array`	The steps' values, in the form :meth:`read` returns.

StoredText `dataclass` ¶source

Bases: TextHandle

Refer to one string packed in a Parquet payload.

The handle holds the reader of the release's payloads and the string's location. A loaded handle also keeps the string in a write-once cell, and read() returns it without payload access. An unloaded handle holds no text, and every read() reads the payload. Handles at one location of one payload file are equal without a read, whether loaded or not.

is_loaded `property` ¶source

is_loaded: bool

Whether the handle keeps the string, so reads need no payload access.

length `property` ¶source

length: int

The string's length, recorded in its location.

store `class-attribute` `instance-attribute` ¶source

store: PackedValueReader = dataclass_field(repr=False)

read ¶source

read() -> str

Return the kept string, or the string from its payload.

StoredTextList `dataclass` ¶source

Bases: TextListHandle

Refer to a list of strings packed in a Parquet payload.

The handle holds the reader of the release's payloads and the list's location. A loaded handle also keeps the strings in a write-once cell, and read() returns a new list of them without payload access. An unloaded handle holds no text, and every read() reads the payload. Handles at one location of one payload file are equal without a read, whether loaded or not.

is_loaded `property` ¶source

is_loaded: bool

Whether the handle keeps the strings, so reads need no payload access.

length `property` ¶source

length: int

The number of strings, recorded in the list's location.

store `class-attribute` `instance-attribute` ¶source

store: PackedValueReader = dataclass_field(repr=False)

read ¶source

read() -> list[str]

Return the kept strings, or the list of strings from its payload, as a new list.

StoredValueColumns `dataclass` ¶source

Name the columns of one public Arrow table whose values can be packed.

fields `instance-attribute` ¶source

fields: Mapping[PackedField, Mapping[PackedValueKind, str]]

Per packed field, the column that holds each value kind.

identity `instance-attribute` ¶source

identity: str

Column holding each row's owner id: a task id, or an annotation content id.

kind `instance-attribute` ¶source

kind: ObjectKind

Object kind whose locators address the packed values.

position `class-attribute` `instance-attribute` ¶source

position: str | None = None

Column holding the ordered position, for a field with several values per object.

materialize_table ¶source

materialize_table(
    table: Table,
    reader: PackedValueReader,
    connection: DuckDBPyConnection,
    columns: StoredValueColumns,
) -> Table

Resolve external text fields while retaining the public Arrow table schema.

Returns:

Type	Description
`Table`	The original table when it has no external values, otherwise a table with native values.

offsets_digest ¶source

offsets_digest(offsets: ndarray | Array) -> bytes

Return the SHA-256 digest of an irregular axis's time offsets.

Every caller hashes the same bytes, contiguous little-endian int64, so one sequence of time offsets has one digest whether it is held as NumPy or as Arrow.

Returns:

Type	Description
`bytes`	The 32-byte digest.

table_writer ¶source

Insert catalog rows using their shared table definitions.

ModelTable `dataclass` ¶source

Bases: Generic[_Row]

Bind validated model rows to one catalog table definition.

definition `instance-attribute` ¶source

definition: Table

write ¶source

write(
    connection: DuckDBPyConnection, rows: Sequence[_Row]
) -> None

Serialize model rows and insert them through the shared catalog writer.

insert_rows ¶source

insert_rows(
    connection: DuckDBPyConnection,
    table: Table,
    rows: Sequence[Mapping[str, object]],
) -> None

Insert one Arrow batch using the catalog table's schema and column order.

Raises:

Type	Description
`TimeFValidationError`	If the rows cannot be converted or inserted into the table.

task_codec ¶source

Canonical codecs between Task fields and the typed columns of the tasks and task_targets tables.

Spans, task configuration, and targets are all stored as plain columns rather than JSON, so DuckDB can filter on them and the reader builds objects straight from row values. Object references are public IDs. The target rows use the task_targets column names, so the rows are stored and read back unchanged. A span holds Signal objects, so encoding reads their ids and decoding looks the read-back Signals up by id.

ANNOTATION_FIELDS `module-attribute` ¶source

ANNOTATION_FIELDS = frozenset(
    {
        TaskRelationshipField.INPUT_ANNOTATIONS,
        TaskRelationshipField.TARGET_ANNOTATIONS,
    }
)

The fields stored in task_annotation_refs.

PAYLOAD_COLUMNS `module-attribute` ¶source

PAYLOAD_COLUMNS = (
    "label_set",
    "unit",
    "target_name",
    "mode",
)

The tasks columns that hold configuration for each task type.

Unused columns are NULL. A classification task stores its label set name in label_set. The label_sets table stores the set itself.

RECORD_FIELDS `module-attribute` ¶source

RECORD_FIELDS = frozenset(
    {
        TaskRelationshipField.INPUTS,
        TaskRelationshipField.CANDIDATE_RECORDS,
    }
)

The fields stored in task_record_refs.

TaskRelationshipField ¶source

Bases: StrEnum

Task field that a task_record_refs or task_annotation_refs row belongs to.

CANDIDATE_RECORDS `class-attribute` `instance-attribute` ¶source

CANDIDATE_RECORDS = 'candidate_records'

INPUTS `class-attribute` `instance-attribute` ¶source

INPUTS = 'inputs'

INPUT_ANNOTATIONS `class-attribute` `instance-attribute` ¶source

INPUT_ANNOTATIONS = 'input_annotations'

TARGET_ANNOTATIONS `class-attribute` `instance-attribute` ¶source

TARGET_ANNOTATIONS = 'target_annotations'

decode_input_modalities ¶source

decode_input_modalities(
    values: list[str] | None,
) -> frozenset[InputModality]

Decode a required stored input modality set.

Returns:

Type	Description
`frozenset[InputModality]`	The declared input kinds.

Raises:

Type	Description
`TimeFFormatError`	If a stored input kind is unknown or repeated.

decode_span ¶source

decode_span(
    span_type: str | None,
    span_start: int | None,
    span_end: int | None,
    signal_ids: Iterable[EntityId] | None,
    *,
    signals: Mapping[EntityId, Signal],
) -> Span | None

Decode the four span columns.

Parameters:

Name	Type	Description	Default
`span_type`	`str \| None`	The stored span shape, or `None` for no span.	required
`span_start`	`int \| None`	The stored start bound.	required
`span_end`	`int \| None`	The stored end bound.	required
`signal_ids`	`Iterable[EntityId] \| None`	The ids of the Signals the span is scoped to.	required
`signals`	`Mapping[EntityId, Signal]`	The read-back Signals by id, which take the place of `signal_ids`.	required

Returns:

Type	Description
`Span \| None`	The decoded Span, or `None` when `span_type` is NULL.

Raises:

Type	Description
`TimeFFormatError`	If the stored span type is unknown, a required bound is NULL, or a Signal id names no read-back Signal.

decode_split ¶source

decode_split(value: str | None) -> Split | None

Decode a stored split.

Returns:

Type	Description
`Split \| None`	The task's partition, or `None` for a task its dataset does not partition.

Raises:

Type	Description
`TimeFFormatError`	If the stored split is unknown.

decode_target ¶source

decode_target(
    row: Mapping[str, object] | TargetValueRow,
    *,
    records: Mapping[EntityId, Record],
    signals: Mapping[EntityId, Signal],
) -> object

Decode one typed target row and resolve stored object references.

Returns:

Type	Description
`object`	The native scalar, Record, Signal, or Span target.

Raises:

Type	Description
`TimeFFormatError`	If a required value or referenced object is missing, or the kind is unknown.

decode_task_payload ¶source

decode_task_payload(
    task_type: TaskType, columns: dict[str, str | None]
) -> dict[str, Any]

Decode concrete Task configuration from the payload columns.

Returns:

Type	Description
`dict[str, Any]`	Keyword arguments for the concrete Task constructor.

encode_span ¶source

encode_span(span: Span | None) -> dict[str, Any] | None

Encode a concrete span as the four span columns without losing its frame or shape.

Returns:

Type	Description
`dict[str, Any] \| None`	`{"span_type", "span_start", "span_end", "signal_ids"}`, or `None`. `signal_ids` is the
`dict[str, Any] \| None`	span's series scope: `None` for every series, a tuple of IDs for a time span, and a
`dict[str, Any] \| None`	one-element tuple for a step span.

Raises:

Type	Description
`TimeFFormatError`	If the Span subtype is unsupported.

encode_task_payload ¶source

encode_task_payload(task: Task) -> dict[str, str | None]

Encode the configuration owned by the concrete Task class as the payload columns.

Returns:

Type	Description
`dict[str, str \| None]`	One value per :data:`PAYLOAD_COLUMNS`, `None` where the task type has no such field.

task_target_kind ¶source

Tags for the scalar, object, and span targets stored in the control catalog.

SPAN_KINDS `module-attribute` ¶source

SPAN_KINDS = frozenset(
    {
        TaskTargetKind.TIME_POINT,
        TaskTargetKind.TIME_INTERVAL,
        TaskTargetKind.STEP_POINT,
        TaskTargetKind.STEP_INTERVAL,
    }
)

The kinds that store a span. tasks.scope_type holds one of these too.

TaskTargetKind ¶source

Bases: StrEnum

Identify the value columns used by one stored task target.

BOOLEAN `class-attribute` `instance-attribute` ¶source

BOOLEAN = 'boolean'

FLOAT `class-attribute` `instance-attribute` ¶source

FLOAT = 'float'

INTEGER `class-attribute` `instance-attribute` ¶source

INTEGER = 'integer'

RECORD `class-attribute` `instance-attribute` ¶source

RECORD = 'record'

SIGNAL `class-attribute` `instance-attribute` ¶source

SIGNAL = 'signal'

STEP_INTERVAL `class-attribute` `instance-attribute` ¶source

STEP_INTERVAL = 'step_interval'

STEP_POINT `class-attribute` `instance-attribute` ¶source

STEP_POINT = 'step_point'

TEXT `class-attribute` `instance-attribute` ¶source

TEXT = 'text'

TIME_INTERVAL `class-attribute` `instance-attribute` ¶source

TIME_INTERVAL = 'time_interval'

TIME_POINT `class-attribute` `instance-attribute` ¶source

TIME_POINT = 'time_point'

is_step_span ¶source

is_step_span(kind: str) -> bool

Return whether a stored target kind uses a single step-indexed Signal.

task_target_row ¶source

Validated target values and catalog rows.

TargetValueRow ¶source

Bases: TimeFModel

Represent a public target with validated values and public object IDs.

boolean_value `class-attribute` `instance-attribute` ¶source

boolean_value: StrictBool | None = None

float_value `class-attribute` `instance-attribute` ¶source

float_value: StrictFloat | None = None

integer_value `class-attribute` `instance-attribute` ¶source

integer_value: StrictInt | None = None

model_config `class-attribute` `instance-attribute` ¶source

model_config = ConfigDict(
    extra="forbid",
    frozen=True,
    revalidate_instances="always",
    serialize_by_alias=True,
    validate_by_name=True,
    validate_default=True,
)

record_id `class-attribute` `instance-attribute` ¶source

record_id: StrictInt | None = None

signal_id `class-attribute` `instance-attribute` ¶source

signal_id: StrictInt | None = None

span_end `class-attribute` `instance-attribute` ¶source

span_end: StrictInt | None = None

span_signal_ids `class-attribute` `instance-attribute` ¶source

span_signal_ids: tuple[StrictInt, ...] | None = None

span_start `class-attribute` `instance-attribute` ¶source

span_start: StrictInt | None = None

target_kind `instance-attribute` ¶source

target_kind: TaskTargetKind

text_value `class-attribute` `instance-attribute` ¶source

text_value: StrictStr | None = None

from_target `classmethod` ¶source

from_target(target: object) -> Self

Construct a validated row for one public target.

Returns:

Type	Description
`Self`	The target's value columns, with unrelated columns left NULL.

Raises:

Type	Description
`TimeFFormatError`	If the target type is unsupported.

Format

timenet.format ¶source

annotation_codec ¶source

VALUE_COLUMNS module-attribute ¶source

VALUE_KINDS module-attribute ¶source

decode_annotation_value ¶source

encode_annotation_value ¶source

annotation_value_kind ¶source

AnnotationValueKind ¶source

BOOLEAN class-attribute instance-attribute ¶source

FLOAT class-attribute instance-attribute ¶source

INTEGER class-attribute instance-attribute ¶source

TEXT class-attribute instance-attribute ¶source

TEXT_LIST class-attribute instance-attribute ¶source

checksums ¶source

CHECKSUM_PREFIX module-attribute ¶source

file_checksum ¶source

stream_checksum ¶source

constants ¶source

DEFAULT_CHUNK_MAX_BYTES module-attribute ¶source

DEFAULT_COMPRESSION module-attribute ¶source

DEFAULT_ROW_GROUP_TARGET_BYTES module-attribute ¶source

DEFAULT_SHARD_TARGET_BYTES module-attribute ¶source

MANIFEST_FILE module-attribute ¶source

MAX_PART_INDEX module-attribute ¶source

PART_INDEX_DIGITS module-attribute ¶source

PAYLOAD_TEMPLATE module-attribute ¶source

SHARD_DIR module-attribute ¶source

SHARD_TEMPLATE module-attribute ¶source

check_relative_path ¶source

part_path ¶source

control_audit ¶source

CROSS_RECORD_PARENT module-attribute ¶source

SOURCE_CYCLE module-attribute ¶source

audit_control_database ¶source

control_cache ¶source

materialize_control ¶source

control_reader ¶source

RECORD_BATCH_SIZE module-attribute ¶source

TASK_BATCH_SIZE module-attribute ¶source

ValueLoaderFactory module-attribute ¶source

DuckDBControlReader ¶source

external instance-attribute ¶source

path instance-attribute ¶source

chunk_location ¶source

chunk_locations_by_ids ¶source

close ¶source

import_map ¶source

imported_ids ¶source

iter_record_batches ¶source

iter_task_batches ¶source

read_annotation_table ¶source

read_dataset_annotations ¶source

read_last_ids ¶source

read_records ¶source

read_signals_by_id ¶source

read_target_table ¶source

read_task_table ¶source

read_tasks ¶source

read_tasks_by_id ¶source

record_clock_ids ¶source

record_ids ¶source

run ¶source

signal_record_ids ¶source

task_facts ¶source

control_schema ¶source

ANNOTATED_OBJECT_TYPES module-attribute ¶source

ANNOTATED_OWNER_TABLES module-attribute ¶source

ANNOTATION_CONTENTS module-attribute ¶source

ANNOTATION_OCCURRENCES module-attribute ¶source

AXES module-attribute ¶source

CLOCKS module-attribute ¶source

CONTROL_METADATA module-attribute ¶source

CONTROL_TABLES module-attribute ¶source

DATASETS module-attribute ¶source

EXTERNAL_OWNER_TABLES module-attribute ¶source

EXTERNAL_VALUES module-attribute ¶source

IMPORTED_OBJECTS module-attribute ¶source

IMPORTS module-attribute ¶source

LABEL_SETS module-attribute ¶source

VALUE_COLUMNS `module-attribute` ¶source

VALUE_KINDS `module-attribute` ¶source

BOOLEAN `class-attribute` `instance-attribute` ¶source

FLOAT `class-attribute` `instance-attribute` ¶source

INTEGER `class-attribute` `instance-attribute` ¶source

TEXT `class-attribute` `instance-attribute` ¶source

TEXT_LIST `class-attribute` `instance-attribute` ¶source

CHECKSUM_PREFIX `module-attribute` ¶source

DEFAULT_CHUNK_MAX_BYTES `module-attribute` ¶source

DEFAULT_COMPRESSION `module-attribute` ¶source

DEFAULT_ROW_GROUP_TARGET_BYTES `module-attribute` ¶source

DEFAULT_SHARD_TARGET_BYTES `module-attribute` ¶source

MANIFEST_FILE `module-attribute` ¶source

MAX_PART_INDEX `module-attribute` ¶source

PART_INDEX_DIGITS `module-attribute` ¶source

PAYLOAD_TEMPLATE `module-attribute` ¶source

SHARD_DIR `module-attribute` ¶source

SHARD_TEMPLATE `module-attribute` ¶source

CROSS_RECORD_PARENT `module-attribute` ¶source

SOURCE_CYCLE `module-attribute` ¶source

RECORD_BATCH_SIZE `module-attribute` ¶source

TASK_BATCH_SIZE `module-attribute` ¶source

ValueLoaderFactory `module-attribute` ¶source

external `instance-attribute` ¶source

path `instance-attribute` ¶source

ANNOTATED_OBJECT_TYPES `module-attribute` ¶source

ANNOTATED_OWNER_TABLES `module-attribute` ¶source

ANNOTATION_CONTENTS `module-attribute` ¶source

ANNOTATION_OCCURRENCES `module-attribute` ¶source

AXES `module-attribute` ¶source

CLOCKS `module-attribute` ¶source

CONTROL_METADATA `module-attribute` ¶source

CONTROL_TABLES `module-attribute` ¶source

DATASETS `module-attribute` ¶source

EXTERNAL_OWNER_TABLES `module-attribute` ¶source

EXTERNAL_VALUES `module-attribute` ¶source

IMPORTED_OBJECTS `module-attribute` ¶source

IMPORTS `module-attribute` ¶source

LABEL_SETS `module-attribute` ¶source

LAST_ID_PREFIX `module-attribute` ¶source

RECORDS `module-attribute` ¶source

SIGNALS `module-attribute` ¶source

SIGNAL_CHUNKS `module-attribute` ¶source

SOURCES `module-attribute` ¶source

TABLES `module-attribute` ¶source

TASKS `module-attribute` ¶source

TASK_ANNOTATION_REFS `module-attribute` ¶source

TASK_DEPENDENCIES `module-attribute` ¶source

TASK_RECORD_REFS `module-attribute` ¶source

TASK_TARGETS `module-attribute` ¶source

Column `dataclass` ¶source

name `instance-attribute` ¶source

nullable `class-attribute` `instance-attribute` ¶source

type `instance-attribute` ¶source

BIGINT `class-attribute` `instance-attribute` ¶source

BIGINT_LIST `class-attribute` `instance-attribute` ¶source

BOOLEAN `class-attribute` `instance-attribute` ¶source

DOUBLE `class-attribute` `instance-attribute` ¶source

JSON `class-attribute` `instance-attribute` ¶source

LABEL_LIST `class-attribute` `instance-attribute` ¶source

VARCHAR `class-attribute` `instance-attribute` ¶source

VARCHAR_LIST `class-attribute` `instance-attribute` ¶source

arrow `instance-attribute` ¶source

sql `instance-attribute` ¶source

ForeignKey `dataclass` ¶source

column `instance-attribute` ¶source

many `class-attribute` `instance-attribute` ¶source

references `instance-attribute` ¶source

table `instance-attribute` ¶source

Table `dataclass` ¶source

arrow_schema `property` ¶source

column_names `property` ¶source

columns `instance-attribute` ¶source

foreign_keys `class-attribute` `instance-attribute` ¶source

name `instance-attribute` ¶source

unique `class-attribute` `instance-attribute` ¶source

external `instance-attribute` ¶source

path `instance-attribute` ¶source

CATALOG_ERRORS `module-attribute` ¶source

CONTROL_FILE `module-attribute` ¶source