# Timenet.format ## timenet.format The TimeF on-disk format contract: filenames, layout templates, and pinned Arrow schemas. The writer and the reader both use this module. They are otherwise independent. The definitions that both sides of the round-trip must agree on live here, so neither side imports the other. ### annotation\_codec Codec between an annotation's value and the typed value columns of `annotation_contents`. An annotation value is a string, an integer, a float, a boolean, a list of strings, or absent for a pure marker. Each shape has its own column so DuckDB can filter on it and the reader hands back the exact Python type that was written: an integer stays an integer and a boolean stays a boolean. #### VALUE\_COLUMNS `module-attribute` ``` VALUE_COLUMNS = ( "value_kind", "text_value", "integer_value", "float_value", "boolean_value", "text_list_value", ) ``` The `annotation_contents` columns that together hold one value. #### VALUE\_KINDS `module-attribute` ``` VALUE_KINDS = tuple( (kind.value) for kind in AnnotationValueKind ) ``` Every `value_kind` a stored annotation may carry. A marker has NULL. #### decode\_annotation\_value ``` decode_annotation_value( value_kind: str | None, text_value: str | None, integer_value: int | None, float_value: float | None, boolean_value: bool | None, text_list_value: list[str] | None, ) -> Any ``` Rebuild one annotation value from its columns. Returns: | Type | Description | | --- | --- | | `Any` | The value in the Python type it was written with, or `None` for a marker. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the kind is unknown or its column is NULL. | #### encode\_annotation\_value ``` encode_annotation_value( value: object, *, label: str ) -> dict[str, Any] ``` Project one annotation value onto the value columns. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `value` | `object` | The annotation's value, or `None` for a marker. | *required* | | `label` | `str` | How to name the annotation in an error message. | *required* | Returns: | Type | Description | | --- | --- | | `dict[str, Any]` | One entry per :data:`VALUE_COLUMNS`, all `None` for a marker. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If the value is not one of the supported shapes. | ### annotation\_value\_kind Tags for annotation values stored in the control catalog. #### AnnotationValueKind Bases: `StrEnum` Identify the populated value column of an annotation's content. ##### BOOLEAN `class-attribute` `instance-attribute` ``` BOOLEAN = 'boolean' ``` ##### FLOAT `class-attribute` `instance-attribute` ``` FLOAT = 'float' ``` ##### INTEGER `class-attribute` `instance-attribute` ``` INTEGER = 'integer' ``` ##### TEXT `class-attribute` `instance-attribute` ``` TEXT = 'text' ``` ##### TEXT\_LIST `class-attribute` `instance-attribute` ``` TEXT_LIST = 'text_list' ``` ### checksums File checksums recorded in the manifest and verified on read. The reader must hash files the same way the writer did, or :meth:`~timenet.reader.TimeFReader.verify` means nothing. The algorithm and the block size live here so both sides call the same code. #### CHECKSUM\_PREFIX `module-attribute` ``` CHECKSUM_PREFIX = 'sha256:' ``` #### file\_checksum ``` file_checksum(path: Path) -> str ``` Return a file's manifest checksum, hashing it a block at a time. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `path` | `Path` | The file to hash. | *required* | Returns: | Type | Description | | --- | --- | | `str` | The checksum as `"sha256:"`. | #### stream\_checksum ``` stream_checksum(handle: _Readable) -> str ``` Return an open binary stream's manifest checksum, hashing it a block at a time. This is the stream counterpart of :func:`file_checksum`. A reader can use it to hash a file opened through a pyarrow filesystem (local now, an object store later) instead of a local `Path`. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `handle` | `_Readable` | An open binary stream positioned at the start. | *required* | Returns: | Type | Description | | --- | --- | | `str` | The checksum as `"sha256:"`. | ### constants On-disk filenames and layout templates for the TimeF format. #### DEFAULT\_CHUNK\_MAX\_BYTES `module-attribute` ``` DEFAULT_CHUNK_MAX_BYTES = 1 * 2 ** 20 ``` #### DEFAULT\_COMPRESSION `module-attribute` ``` DEFAULT_COMPRESSION = 'zstd' ``` #### DEFAULT\_ROW\_GROUP\_TARGET\_BYTES `module-attribute` ``` DEFAULT_ROW_GROUP_TARGET_BYTES = 4 * 2 ** 20 ``` #### DEFAULT\_SHARD\_TARGET\_BYTES `module-attribute` ``` DEFAULT_SHARD_TARGET_BYTES = 128 * 2 ** 20 ``` #### MANIFEST\_FILE `module-attribute` ``` MANIFEST_FILE = 'manifest.json' ``` #### MAX\_PART\_INDEX `module-attribute` ``` MAX_PART_INDEX = 10 ** PART_INDEX_DIGITS - 1 ``` #### PART\_INDEX\_DIGITS `module-attribute` ``` PART_INDEX_DIGITS = 8 ``` #### PAYLOAD\_TEMPLATE `module-attribute` ``` PAYLOAD_TEMPLATE = 'payloads/part-{:08d}.parquet' ``` #### SHARD\_DIR `module-attribute` ``` SHARD_DIR = 'time_series' ``` #### SHARD\_TEMPLATE `module-attribute` ``` SHARD_TEMPLATE = 'time_series/part-{:08d}.parquet' ``` #### check\_relative\_path ``` check_relative_path(name: str, path: str) -> str ``` Reject a path that would resolve outside the dataset root it is meant to stay under. Callers join a manifest file entry and a registry handle's `relpath` onto a root directory (or filesystem prefix) as-is. An absolute path or a `..` segment would escape that root instead of raising. This would let a crafted manifest or caller read or write outside the intended directory. Stored paths use forward slashes on every platform; Windows drive paths and backslashes are rejected even when validation runs on another operating system. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `name` | `str` | The name of the field to check. It appears in the error message. | *required* | | `path` | `str` | The path to check. It must be relative and rooted inside the dataset. | *required* | Returns: | Type | Description | | --- | --- | | `str` | The validated path. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If `path` is absolute, names a Windows drive, contains backslashes, or has a `..` segment. | #### part\_path ``` part_path(template: str, index: int, **fields: str) -> str ``` Render a numbered part/shard path, refusing an index that would overflow the fixed width. The reader visits parts in the order the manifest lists them. The zero-padded names keep an `ls` or a prefix listing in that same order. That order only holds while every number fits in `PART_INDEX_DIGITS`. A ninth digit sorts before the eighth and silently breaks it. Routing every part name through here turns that overflow into a loud error instead. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `template` | `str` | A layout template whose part number is the `{:08d}` field, for example `SHARD_TEMPLATE`. | *required* | | `index` | `int` | The zero-based part number to render. | *required* | | `fields` | `str` | Any remaining named fields the template needs, for example `task_type`. | `{}` | Returns: | Type | Description | | --- | --- | | `str` | The formatted version-relative path. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If `index` is negative or needs more than `PART_INDEX_DIGITS` digits. | ### control\_audit Audit a finished `control.duckdb` for relational integrity. The control database has no persisted constraints. ART indexes on large tables such as `signal_chunks` make every write slower and the file larger. Instead, the writer keeps the invariants while it streams rows. It checks them in Python with ID sets. This module holds the SQL form of the same invariants. It scans every table, so it is not part of the write path. Tests run it after each round trip, and tooling can run it on a published version. #### CROSS\_RECORD\_PARENT `module-attribute` ``` CROSS_RECORD_PARENT = "\n SELECT child.source_id\n FROM sources child\n JOIN sources parent ON parent.source_id = child.parent_source_id\n WHERE child.record_id <> parent.record_id\n LIMIT 1\n" ``` #### SOURCE\_CYCLE `module-attribute` ``` SOURCE_CYCLE = "\n WITH RECURSIVE ancestors(source_id, parent_source_id, path, cyclic) AS (\n SELECT source_id, parent_source_id, [source_id], false FROM sources\n UNION ALL\n SELECT parent.source_id, parent.parent_source_id,\n list_append(ancestors.path, parent.source_id),\n list_contains(ancestors.path, parent.source_id)\n FROM ancestors\n JOIN sources parent ON parent.source_id = ancestors.parent_source_id\n WHERE NOT ancestors.cyclic\n )\n SELECT source_id FROM ancestors WHERE cyclic LIMIT 1\n" ``` #### audit\_control\_database ``` audit_control_database( connection: DuckDBPyConnection, *, require_chunks: bool = True, ) -> None ``` Scan every control table and raise an error at the first broken rule. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `connection` | `DuckDBPyConnection` | An open connection to the control database. | *required* | | `require_chunks` | `bool` | Whether every Signal must be covered exactly by `signal_chunks` rows. A metadata-only database written without value placements has none. | `True` | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If a key repeats, a reference points to a missing object, or a value is not allowed. Also if Sources do not form separate trees for each Record. Also if an axis or chunk length differs from its Signal. Also if classification labels break a rule of their label set. | ### control\_cache Materialize a verified DuckDB control file behind a local-path seam. #### materialize\_control ``` materialize_control(version: DatasetVersion) -> Path ``` Return a local path for a version's verified DuckDB control database. Local versions already satisfy DuckDB's path requirement. For a remote filesystem, this function downloads only `control.duckdb` into a checksum-keyed cache. A future remote-attach implementation can replace this function without changing the schema or query layer. Returns: | Type | Description | | --- | --- | | `Path` | A local path suitable for a read-only DuckDB connection. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the manifest lacks a control file or the downloaded file fails verification. | ### control\_reader Build the TimeF object hierarchy from `control.duckdb`. The objects hold their stored values behind value handles, unloaded. #### RECORD\_BATCH\_SIZE `module-attribute` ``` RECORD_BATCH_SIZE = 1000 ``` The most Records one batch of a Record stream holds, unless the stream sets another batch size. #### TASK\_BATCH\_SIZE `module-attribute` ``` TASK_BATCH_SIZE = 5000 ``` The most tasks one batch of a task read or stream holds, unless a stream sets another batch size. #### ValueLoaderFactory `module-attribute` ``` ValueLoaderFactory = Callable[ [int, TimeSeriesSpec, int], ArrayHandle ] ``` #### DuckDBControlReader Keep one read-only DuckDB connection and build requested records from it. The connection opens again on the next query after :meth:`close`, and after the process forks, since every connection closes before a fork. ##### external `instance-attribute` ``` external = external ``` ##### path `instance-attribute` ``` path = Path(path) ``` ##### chunk\_location ``` chunk_location(signal_id: int) -> ChunkLocation ``` Return where one Signal's chunks are stored, from its catalog rows. Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the Signal has no stored chunks. | ##### chunk\_locations\_by\_ids ``` chunk_locations_by_ids( signal_ids: Iterable[int], ) -> dict[int, ChunkLocation | None] ``` Return where a batch of Signals' chunks are stored, from their catalog rows. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `signal_ids` | `Iterable[int]` | Signal IDs. | *required* | Returns: | Type | Description | | --- | --- | | `dict[int, ChunkLocation | None]` | Locations keyed by Signal ID, each with the Signal's chunks in chunk-index order, or | | `dict[int, ChunkLocation | None]` | `None` for a Signal without stored chunks. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If one Signal's chunks are stored by more than one release. | ##### close ``` close() -> None ``` Close the reader's DuckDB connection and drop its cached catalog facts. ##### import\_map ``` import_map() -> tuple[dict[str, str], ImportMap] ``` Return the parent release of each parent alias, and the import map of the catalog. Returns: | Type | Description | | --- | --- | | `dict[str, str]` | The parent release that `imports` names for each parent alias, and the local ID of each | | `ImportMap` | object in `imported_objects`, keyed by parent alias, kind, and parent ID. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the catalog lists a parent alias, or a parent ID under one alias and kind, more than once. | ##### imported\_ids ``` imported_ids( alias: str, kind: ObjectKind, parent_ids: Iterable[EntityId], ) -> tuple[str | None, dict[EntityId, EntityId]] ``` Return the parent release of a parent alias and the local IDs of objects imported under it. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `alias` | `str` | The parent alias the objects were imported under. | *required* | | `kind` | `ObjectKind` | The kind of the objects. | *required* | | `parent_ids` | `Iterable[EntityId]` | The objects' IDs in the parent release. | *required* | Returns: | Type | Description | | --- | --- | | `str | None` | The parent release the catalog names for the alias, or `None` if it names none, and | | `dict[EntityId, EntityId]` | the local object ID of each parent ID the catalog knows. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the catalog lists the alias more than once, or lists one of the parent IDs more than once under the alias and kind. | ##### iter\_record\_batches ``` iter_record_batches( record_ids: Iterable[int] | None = None, *, with_annotations: bool = True, batch_size: int = RECORD_BATCH_SIZE, ) -> Iterator[tuple[Record, ...]] ``` Yield complete records one bounded batch at a time, and build each batch on its own. Without `record_ids`, the stream reads the Record IDs one page at a time, in read order. A page holds up to `batch_size` IDs. The stream builds one batch from each page that is not empty. The next page starts after the last read order of the page before. The stream stops after the first page that holds fewer than `batch_size` IDs. So the stream does not read all Record IDs before the first batch. With `record_ids`, the stream checks every requested ID before it builds the first batch. Each batch reads only its own rows. The records of a batch share no object with another batch. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `record_ids` | `Iterable[int] | None` | Requested IDs in result order, or `None` for every record in read order. | `None` | | `with_annotations` | `bool` | Whether to build annotations on records, sources, and signals. | `True` | | `batch_size` | `int` | The most records that a batch holds. By default, a batch holds up to 1,000 records (`RECORD_BATCH_SIZE`). | `RECORD_BATCH_SIZE` | Yields: | Type | Description | | --- | --- | | `tuple[Record, ...]` | The records of each batch, with their stored values behind unloaded handles. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If `batch_size` is not a positive int, or if a requested ID is not an int. | | `TimeFNotFoundError` | If a requested record is absent. | ##### iter\_task\_batches ``` iter_task_batches( records: Iterable[Record] | None = None, *, required_modalities: Iterable[InputModality] | None = None, supported_modalities: Iterable[InputModality] | None = None, split: Split | str | None = None, batch_size: int = TASK_BATCH_SIZE, ) -> Iterator[list[Task]] ``` Yield the tasks whose inputs include one of `records`, a bounded batch at a time. Each batch runs a handful of keyed queries and builds only that batch's tasks, so a caller that walks one Record at a time never holds the whole task table in memory. Only `None` skips the input Record filter; an empty collection selects no tasks. Each batch is built on its own. The parents its tasks derive from through `from_tasks`, and the Records, Signals, and annotations they refer to outside `records`, are built for the batch and shared only among its tasks, so nothing of a yielded batch stays with the stream. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `records` | `Iterable[Record] | None` | Input Records to select, or `None` for every task. Tasks reuse these objects. | `None` | | `split` | `Split | str | None` | Keep only tasks of this split, or `None` for every task. | `None` | | `batch_size` | `int` | The most tasks a batch holds. By default, a batch holds up to 5,000 tasks (`TASK_BATCH_SIZE`). Pass another value for larger or smaller batches. | `TASK_BATCH_SIZE` | Yields: | Type | Description | | --- | --- | | `list[Task]` | Each batch's concrete tasks, in stable ID order across batches. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If a relationship refers to a missing object or has an invalid field. | ##### read\_annotation\_table ``` read_annotation_table( object_type: ObjectKind | None = None, ) -> Table ``` Return every annotation occurrence joined with its content as Arrow. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `object_type` | `ObjectKind | None` | Keep only occurrences on this object type, or `None` for all. | `None` | Returns: | Type | Description | | --- | --- | | `Table` | One row per occurrence. Each row has the annotated object's type and ID, the content's | | `Table` | ID and key, typed value columns, unit, span columns, confidence, and JSON metadata. The | | `Table` | object ID is NULL for the Dataset, which has a string `dataset_id` instead. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If `object_type` is not an annotated object type. | ##### read\_dataset\_annotations ``` read_dataset_annotations( dataset_id: str, *, by_occurrence: _AnnotationsByOccurrence | None = None, ) -> tuple[Annotation, ...] ``` Return annotations attached directly to a dataset, as new objects on every call. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `dataset_id` | `str` | The dataset whose annotations to read. | *required* | | `by_occurrence` | `_AnnotationsByOccurrence | None` | Filled with every annotation by its occurrence id, so that a task read of the same call can reference these objects. | `None` | Returns: | Type | Description | | --- | --- | | `tuple[Annotation, ...]` | Dataset-level annotation occurrences in storage order. | ##### read\_last\_ids ``` read_last_ids() -> dict[ObjectKind, int] ``` Return the last used id per entity kind, as the writer stored it. Returns: | Type | Description | | --- | --- | | `dict[ObjectKind, int]` | The stored counters. The result does not include a kind without a stored value. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If a stored counter names an unknown kind or is not an integer. | ##### read\_records ``` read_records( record_ids: Iterable[int] | None = None, *, with_annotations: bool = True, shared_offsets: dict[EntityId, StoredOffsets] | None = None, ) -> tuple[Record, ...] ``` Build complete recursive records, with their stored values behind unloaded handles. Source links, clocks, axes, and time-offset locations are checked before any object is built. No payload file is opened. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `record_ids` | `Iterable[int] | None` | Requested IDs in result order, or `None` for every record in read order. | `None` | | `with_annotations` | `bool` | Whether to build annotations on records, sources, and signals. | `True` | | `shared_offsets` | `dict[EntityId, StoredOffsets] | None` | Time-offset handles by axis ID, shared with earlier record reads of the same call, and extended with this read's new ones. `None` shares nothing. | `None` | Returns: | Type | Description | | --- | --- | | `tuple[Record, ...]` | The built records. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If a requested ID is not an int. | | `TimeFNotFoundError` | If a requested record is absent. | | `TimeFFormatError` | If the stored hierarchy is inconsistent. | ##### read\_signals\_by\_id ``` read_signals_by_id( signal_ids: Collection[EntityId], ) -> dict[EntityId, Signal] ``` Build the Records that hold the given Signals, and return those Signals. Each Record is built once, however many of its Signals are asked for, and the Records of one call share one time-offsets handle per axis. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `signal_ids` | `Collection[EntityId]` | The Signals' IDs. | *required* | Returns: | Type | Description | | --- | --- | | `dict[EntityId, Signal]` | The Signals by ID, each inside its Record, with stored values behind unloaded handles. | Raises: | Type | Description | | --- | --- | | `TimeFNotFoundError` | If an ID names no stored Signal. | ##### read\_target\_table ``` read_target_table( record_ids: Iterable[int] | None = None, *, required_modalities: Iterable[InputModality] | None = None, supported_modalities: Iterable[InputModality] | None = None, split: Split | str | None = None, ) -> Table ``` Return every inline target as Arrow, one typed row each, with references as public IDs. The columns follow :class:`timenet.format.task_target_row.TargetValueRow`: a step span names its one series in `span_signal_ids`. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `record_ids` | `Iterable[int] | None` | Keep only targets of tasks whose inputs include one of these Records. | `None` | | `split` | `Split | str | None` | Keep only targets of tasks of this split, or `None` for all. | `None` | Returns: | Type | Description | | --- | --- | | `Table` | Targets in task ID and position order. | ##### read\_task\_table ``` read_task_table( record_ids: Iterable[int] | None = None, *, required_modalities: Iterable[InputModality] | None = None, supported_modalities: Iterable[InputModality] | None = None, split: Split | str | None = None, ) -> Table ``` Return the task table as Arrow, with object references as public IDs, without building Tasks. The table has one row per task. Each row holds the type, prompt, rationale, the four scope columns, the configuration columns, ordered `input_record_ids`, ordered `from_task_ids`, and the metadata JSON string. Targets are in :meth:`read_target_table`. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `record_ids` | `Iterable[int] | None` | Keep only tasks whose inputs include one of these Records, or `None` for all. | `None` | | `split` | `Split | str | None` | Keep only tasks of this split, or `None` for all. | `None` | Returns: | Type | Description | | --- | --- | | `Table` | The tasks in task ID order. | ##### read\_tasks ``` read_tasks( records: Iterable[Record] | None = None, *, cached_records: Iterable[Record] = (), dataset_annotations: Mapping[int, Annotation] | None = None, required_modalities: Iterable[InputModality] | None = None, supported_modalities: Iterable[InputModality] | None = None, split: Split | str | None = None, ) -> tuple[Task, ...] ``` Build every selected task and restore all in-memory object references. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `records` | `Iterable[Record] | None` | Input Records to select, or `None` for every task. Tasks reuse these objects. | `None` | | `cached_records` | `Iterable[Record]` | Records already built, to reuse without restricting the selection. | `()` | | `dataset_annotations` | `Mapping[int, Annotation] | None` | The dataset's annotations, already built, by occurrence id, which tasks then reference instead of their own copies, or `None` to read them when a task needs them. | `None` | | `split` | `Split | str | None` | Keep only tasks of this split, or `None` for every task. | `None` | Returns: | Type | Description | | --- | --- | | `tuple[Task, ...]` | Concrete tasks in stable ID order. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If a relationship refers to a missing object or has an invalid field. | ##### read\_tasks\_by\_id ``` read_tasks_by_id( task_ids: Collection[EntityId], ) -> dict[EntityId, Task] ``` Build the given tasks, sharing the Records they reference, a bounded batch at a time. The tasks they derive from through `from_tasks` are built too. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `task_ids` | `Collection[EntityId]` | The task IDs, each once. | *required* | Returns: | Type | Description | | --- | --- | | `dict[EntityId, Task]` | The tasks by ID. | Raises: | Type | Description | | --- | --- | | `TimeFNotFoundError` | If an ID names no stored task. | ##### record\_clock\_ids ``` record_clock_ids( record_ids: Collection[EntityId], ) -> dict[EntityId, int] ``` Return the clock of each of the given Records, building no object. A clock ID names one clock within this catalog only. Records with one clock ID share one `TimeOrigin` when a read builds them. Returns: | Type | Description | | --- | --- | | `dict[EntityId, int]` | The clock ID of each Record, by Record ID. An ID that names no Record is absent. | ##### record\_ids ``` record_ids() -> tuple[int, ...] ``` Return every stored Record ID in the order :meth:`read_records` yields them, without building a Record. Returns: | Type | Description | | --- | --- | | `tuple[int, ...]` | The Record IDs in their stored read order. | ##### run ``` run(query: Callable[[DuckDBPyConnection], _T]) -> _T ``` Run one read through the connection, which opens again first if it was closed. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `query` | `Callable[[DuckDBPyConnection], _T]` | Reads through the connection it is given. | *required* | Returns: | Type | Description | | --- | --- | | `_T` | What `query` returns. | Raises: | Type | Description | | --- | --- | | `TimeFReaderError` | If the connection closed while the read ran, through :meth:`close` or through the before-fork hook in another thread. | | `TimeFFormatError` | If the catalog lacks a table, column, or type that the read expects. | | `Error` | Any other DuckDB failure, such as I/O, memory, or an interrupt, unchanged. | ##### signal\_record\_ids ``` signal_record_ids( signal_ids: Collection[EntityId], ) -> dict[EntityId, EntityId] ``` Return the ID of the Record that holds each of the given Signals, building no object. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `signal_ids` | `Collection[EntityId]` | The Signals' IDs. | *required* | Returns: | Type | Description | | --- | --- | | `dict[EntityId, EntityId]` | The Record ID of each Signal, by Signal ID. | Raises: | Type | Description | | --- | --- | | `TimeFNotFoundError` | If an ID names no stored Signal. | ##### task\_facts ``` task_facts() -> _TaskFacts ``` Return the per-database facts that task reads need, read once per reader. Returns: | Type | Description | | --- | --- | | `_TaskFacts` | The IDs of tasks other tasks derive from, and whether any task carries annotations. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If two task rows share an ID. | ### control\_schema The single description of the control database's tables. The SQL table definitions and the writer's Arrow batch schemas come from :data:`CONTROL_TABLES`. The audit also uses it to examine keys and references. Define schema changes there. This is version 2 of the control schema. Every table except `label_sets` uses ids as keys. Records, Signals, and Tasks use the public ids that the caller sees. All other objects use the writer's storage ids. Each label set has a unique name in a dataset. The `label_sets` table uses this name as its key. A foreign key stores the key of the row that it refers to. Fixed-shape fields are typed columns, so SQL can filter them. Only the free-form `metadata` dictionaries are JSON. The database has no persisted constraints. The writer keeps :data:`Table.unique` and :data:`Table.foreign_keys` while it streams rows, and :func:`timenet.format.control_audit.audit_control_database` checks them again. #### ANNOTATED\_OBJECT\_TYPES `module-attribute` ``` ANNOTATED_OBJECT_TYPES: frozenset[ObjectKind] = frozenset( ANNOTATED_OWNER_TABLES ) ``` The values `annotation_occurrences.object_type` can hold. #### ANNOTATED\_OWNER\_TABLES `module-attribute` ``` ANNOTATED_OWNER_TABLES: dict[ ObjectKind, tuple[str, str] | None ] = { ObjectKind.DATASET: None, ObjectKind.RECORD: ("records", "record_id"), ObjectKind.SOURCE: ("sources", "source_id"), ObjectKind.SIGNAL: ("signals", "signal_id"), ObjectKind.TASK: ("tasks", "task_id"), } ``` The owner table and key column of each annotated object type. The Dataset has no owner row. #### ANNOTATION\_CONTENTS `module-attribute` ``` ANNOTATION_CONTENTS = Table( "annotation_contents", ( _required("content_id", ColumnType.BIGINT), _required("name"), Column("value_kind", ColumnType.VARCHAR), Column("text_value", ColumnType.VARCHAR), Column("integer_value", ColumnType.BIGINT), Column("float_value", ColumnType.DOUBLE), Column("boolean_value", ColumnType.BOOLEAN), Column("text_list_value", ColumnType.VARCHAR_LIST), Column("unit", ColumnType.VARCHAR), Column("description", ColumnType.VARCHAR), _required("metadata", ColumnType.JSON), ), unique=(("content_id",),), ) ``` #### ANNOTATION\_OCCURRENCES `module-attribute` ``` ANNOTATION_OCCURRENCES = Table( "annotation_occurrences", ( _required("occurrence_id", ColumnType.BIGINT), _required("content_id", ColumnType.BIGINT), _required("object_type"), Column("object_id", ColumnType.BIGINT), _required("span_type"), Column("start_us", ColumnType.BIGINT), Column("end_us", ColumnType.BIGINT), Column("provenance", ColumnType.JSON), Column("confidence", ColumnType.DOUBLE), _required("metadata", ColumnType.JSON), ), unique=(("occurrence_id",),), foreign_keys=( ForeignKey( "content_id", "annotation_contents", "content_id", ), ), ) ``` The owner is `(object_type, object_id)`. `object_id` is the public id of a Record, Signal, or Task, or the storage id of a Source. Ids are unique only within one kind, so the type is part of the reference. `object_id` is NULL exactly when the owner is the Dataset. #### AXES `module-attribute` ``` AXES = Table( "axes", ( _required("axis_id", ColumnType.BIGINT), _required("axis_type"), Column("period_numerator_us", ColumnType.BIGINT), Column("period_denominator", ColumnType.BIGINT), Column("start_index", ColumnType.BIGINT), Column("offset_us", ColumnType.BIGINT), Column("first_us", ColumnType.BIGINT), Column("last_us", ColumnType.BIGINT), ), unique=(("axis_id",),), ) ``` #### CLOCKS `module-attribute` ``` CLOCKS = Table( "clocks", ( _key("clock_id"), Column("start_time_us", ColumnType.BIGINT), ), unique=(("clock_id",),), ) ``` #### CONTROL\_METADATA `module-attribute` ``` CONTROL_METADATA = Table( "control_metadata", (_required("key"), _required("value")), unique=(("key",),), ) ``` #### CONTROL\_TABLES `module-attribute` ``` CONTROL_TABLES: tuple[Table, ...] = ( CONTROL_METADATA, IMPORTS, EXTERNAL_VALUES, IMPORTED_OBJECTS, DATASETS, RECORDS, CLOCKS, SOURCES, AXES, SIGNALS, SIGNAL_CHUNKS, ANNOTATION_CONTENTS, ANNOTATION_OCCURRENCES, LABEL_SETS, TASKS, TASK_TARGETS, TASK_RECORD_REFS, TASK_ANNOTATION_REFS, TASK_DEPENDENCIES, ) ``` Every control table, in creation order. #### DATASETS `module-attribute` ``` DATASETS = Table( "datasets", (_required("dataset_id"),), unique=(("dataset_id",),), ) ``` Exactly one row, the dataset the file describes. #### EXTERNAL\_OWNER\_TABLES `module-attribute` ``` EXTERNAL_OWNER_TABLES: dict[ObjectKind, tuple[str, str]] = { ObjectKind.TASK: ("tasks", "task_id"), ObjectKind.ANNOTATION_CONTENT: ( "annotation_contents", "content_id", ), ObjectKind.AXIS: ("axes", "axis_id"), } ``` The owner table and key column of each object kind whose values can be stored externally. #### EXTERNAL\_VALUES `module-attribute` ``` EXTERNAL_VALUES = Table( "external_values", ( _required("object_kind"), _required("object_id", ColumnType.BIGINT), _required("field"), _key("position"), Column("source_key", ColumnType.VARCHAR), _required("value_path"), _key("row_index"), _required("value_kind"), _key("n_values"), Column("value_digest", ColumnType.VARCHAR), ), unique=( ("object_kind", "object_id", "field", "position"), ), ) ``` The owner is `(object_kind, object_id)`: a Task by its public id, or an annotation content row or an irregular axis by the storage id that this write gave its row. Ids are unique only within one kind, so the kind is part of the reference. Signals that share an axis share its row, so the axis's time offsets are stored once. #### IMPORTED\_OBJECTS `module-attribute` ``` IMPORTED_OBJECTS = Table( "imported_objects", ( _required("alias"), _required("object_kind"), _required("parent_id", ColumnType.BIGINT), _required("object_id", ColumnType.BIGINT), ), unique=(("alias", "object_kind", "parent_id"),), foreign_keys=(ForeignKey("alias", "imports", "alias"),), ) ``` Local object IDs indexed by parent alias, object kind, and original parent ID. #### IMPORTS `module-attribute` ``` IMPORTS = Table( "imports", (_required("alias"), _required("parent_ref")), unique=(("alias",), ("parent_ref",)), ) ``` Catalog lookup projection of the aliases and exact releases in `Manifest.parent_cards`. #### LABEL\_SETS `module-attribute` ``` LABEL_SETS = Table( "label_sets", ( _required("name"), _required("labels", ColumnType.LABEL_LIST), _required("multi_label", ColumnType.BOOLEAN), ), unique=(("name",),), ) ``` Each row stores one label set that a classification task uses. The `labels` column holds one `{name, description}` entry per label, in label order. If a label has no description, `description` is NULL. Tasks refer to a set by its name in `tasks.label_set`. #### LAST\_ID\_PREFIX `module-attribute` ``` LAST_ID_PREFIX = 'last_id.' ``` Prefix of the `control_metadata` keys that store the last id used per entity kind. #### RECORDS `module-attribute` ``` RECORDS = Table( "records", ( _required("record_id", ColumnType.BIGINT), _key("clock_id"), Column("time_span_start_us", ColumnType.BIGINT), Column("time_span_end_us", ColumnType.BIGINT), _required("metadata", ColumnType.JSON), _required("read_order", ColumnType.BIGINT), ), unique=(("record_id",), ("read_order",)), foreign_keys=( ForeignKey("clock_id", "clocks", "clock_id"), ), ) ``` #### SIGNALS `module-attribute` ``` SIGNALS = Table( "signals", ( _required("signal_id", ColumnType.BIGINT), _key("source_id"), _required("name"), _key("axis_id"), _required("spec_type"), _required("spec_name"), Column("unit", ColumnType.VARCHAR), _required("dtype"), _required("categories", ColumnType.VARCHAR_LIST), _required("value_shape", ColumnType.BIGINT_LIST), _required( "dimension_names", ColumnType.VARCHAR_LIST ), Column("missing_reason", ColumnType.VARCHAR), Column("missing_description", ColumnType.VARCHAR), _key("n_values"), Column("null_count", ColumnType.BIGINT), _required("metadata", ColumnType.JSON), _required("modality"), ), unique=(("signal_id",),), foreign_keys=( ForeignKey("source_id", "sources", "source_id"), ForeignKey("axis_id", "axes", "axis_id"), ), ) ``` The `missing_*` columns hold a spec's missing-value record, or are all null when it has none. A specification allows missing timesteps when `missing_reason` is set. `null_count` is the number of missing timesteps among the signal's `n_values` timesteps. It is null only when the signal has no stored values and no supplied count. #### SIGNAL\_CHUNKS `module-attribute` ``` SIGNAL_CHUNKS = Table( "signal_chunks", ( _key("signal_id"), _key("chunk_index"), _required("value_path"), Column("source_key", ColumnType.VARCHAR), _key("chunk_major_index"), Column("chunk_minor_index", ColumnType.BIGINT), _key("n_values"), ), unique=(("signal_id", "chunk_index"),), foreign_keys=( ForeignKey("signal_id", "signals", "signal_id"), ), ) ``` #### SOURCES `module-attribute` ``` SOURCES = Table( "sources", ( _required("source_id", ColumnType.BIGINT), _key("record_id"), Column("parent_source_id", ColumnType.BIGINT), _required("name"), _required("metadata", ColumnType.JSON), ), unique=(("source_id",),), foreign_keys=( ForeignKey("record_id", "records", "record_id"), ForeignKey( "parent_source_id", "sources", "source_id" ), ), ) ``` #### TABLES `module-attribute` ``` TABLES: dict[str, Table] = { (table.name): table for table in CONTROL_TABLES } ``` The control tables by name. #### TASKS `module-attribute` ``` TASKS = Table( "tasks", ( _required("task_id", ColumnType.BIGINT), _required("task_type"), Column("prompt", ColumnType.VARCHAR), _required( "input_modalities", ColumnType.VARCHAR_LIST ), Column("split", ColumnType.VARCHAR), Column("scope_type", ColumnType.VARCHAR), Column("scope_start", ColumnType.BIGINT), Column("scope_end", ColumnType.BIGINT), Column("scope_signal_ids", ColumnType.BIGINT_LIST), _required("has_inline_targets", ColumnType.BOOLEAN), Column("label_set", ColumnType.VARCHAR), Column("unit", ColumnType.VARCHAR), Column("target_name", ColumnType.VARCHAR), Column("mode", ColumnType.VARCHAR), Column("rationale", ColumnType.VARCHAR), _required("metadata", ColumnType.JSON), ), unique=(("task_id",),), foreign_keys=( ForeignKey( "scope_signal_ids", "signals", "signal_id", many=True, ), ForeignKey("label_set", "label_sets", "name"), ), ) ``` #### TASK\_ANNOTATION\_REFS `module-attribute` ``` TASK_ANNOTATION_REFS = Table( "task_annotation_refs", ( _key("task_id"), _required("field"), _key("position"), _key("occurrence_id"), ), unique=(("task_id", "field", "position"),), foreign_keys=( ForeignKey("task_id", "tasks", "task_id"), ForeignKey( "occurrence_id", "annotation_occurrences", "occurrence_id", ), ), ) ``` #### TASK\_DEPENDENCIES `module-attribute` ``` TASK_DEPENDENCIES = Table( "task_dependencies", ( _key("task_id"), _key("position"), _key("parent_task_id"), ), unique=(("task_id", "position"),), foreign_keys=( ForeignKey("task_id", "tasks", "task_id"), ForeignKey("parent_task_id", "tasks", "task_id"), ), ) ``` #### TASK\_RECORD\_REFS `module-attribute` ``` TASK_RECORD_REFS = Table( "task_record_refs", ( _key("task_id"), _required("field"), _key("position"), _key("record_id"), ), unique=(("task_id", "field", "position"),), foreign_keys=( ForeignKey("task_id", "tasks", "task_id"), ForeignKey("record_id", "records", "record_id"), ), ) ``` #### TASK\_TARGETS `module-attribute` ``` TASK_TARGETS = Table( "task_targets", ( _key("task_id"), _key("position"), _required("target_kind"), Column("text_value", ColumnType.VARCHAR), Column("integer_value", ColumnType.BIGINT), Column("float_value", ColumnType.DOUBLE), Column("boolean_value", ColumnType.BOOLEAN), Column("record_id", ColumnType.BIGINT), Column("signal_id", ColumnType.BIGINT), Column("span_start", ColumnType.BIGINT), Column("span_end", ColumnType.BIGINT), Column("span_signal_ids", ColumnType.BIGINT_LIST), ), unique=(("task_id", "position"),), foreign_keys=( ForeignKey("task_id", "tasks", "task_id"), ForeignKey("record_id", "records", "record_id"), ForeignKey("signal_id", "signals", "signal_id"), ForeignKey( "span_signal_ids", "signals", "signal_id", many=True, ), ), ) ``` The value columns match :class:`timenet.format.task_target_row.TargetValueRow`. A span names its series in `span_signal_ids`: one Signal for a step span, and any number for a time span. #### Column `dataclass` One column of a control table. ##### name `instance-attribute` ``` name: str ``` ##### nullable `class-attribute` `instance-attribute` ``` nullable: bool = True ``` ##### type `instance-attribute` ``` type: ColumnType ``` ##### ddl ``` ddl() -> str ``` Return the column's `CREATE TABLE` fragment. Returns: | Type | Description | | --- | --- | | `str` | The column definition without a trailing comma. | #### ColumnType Bases: `Enum` The DuckDB column types the control schema uses, each with the Arrow type that feeds it. The writer builds every batch with the table's Arrow schema, so a value of the wrong Python type fails in Arrow, naming the column, instead of being cast by DuckDB. ##### BIGINT `class-attribute` `instance-attribute` ``` BIGINT = ('BIGINT', pa.int64()) ``` ##### BIGINT\_LIST `class-attribute` `instance-attribute` ``` BIGINT_LIST = ('BIGINT[]', pa.list_(pa.int64())) ``` ##### BOOLEAN `class-attribute` `instance-attribute` ``` BOOLEAN = ('BOOLEAN', pa.bool_()) ``` ##### DOUBLE `class-attribute` `instance-attribute` ``` DOUBLE = ('DOUBLE', pa.float64()) ``` ##### JSON `class-attribute` `instance-attribute` ``` JSON = ('JSON', pa.string()) ``` ##### LABEL\_LIST `class-attribute` `instance-attribute` ``` LABEL_LIST = ( "STRUCT(name VARCHAR, description VARCHAR)[]", pa.list_( pa.struct( [ ("name", pa.string()), ("description", pa.string()), ] ) ), ) ``` ##### VARCHAR `class-attribute` `instance-attribute` ``` VARCHAR = ('VARCHAR', pa.string()) ``` ##### VARCHAR\_LIST `class-attribute` `instance-attribute` ``` VARCHAR_LIST = ('VARCHAR[]', pa.list_(pa.string())) ``` ##### arrow `instance-attribute` ``` arrow = arrow ``` The Arrow type of the column in a writer batch. ##### sql `instance-attribute` ``` sql = sql ``` The type name in `CREATE TABLE`. #### ForeignKey `dataclass` A column whose non-NULL values must exist in another table's column. ##### column `instance-attribute` ``` column: str ``` ##### many `class-attribute` `instance-attribute` ``` many: bool = False ``` Whether `column` is a list column whose every element must exist. ##### references `instance-attribute` ``` references: str ``` ##### table `instance-attribute` ``` table: str ``` #### Table `dataclass` One control table: its columns in storage order, and the invariants it holds. ##### arrow\_schema `property` ``` arrow_schema: Schema ``` The schema of a writer batch for this table, one Arrow field per column. ##### column\_names `property` ``` column_names: tuple[str, ...] ``` The column names in storage order. ##### columns `instance-attribute` ``` columns: tuple[Column, ...] ``` ##### foreign\_keys `class-attribute` `instance-attribute` ``` foreign_keys: tuple[ForeignKey, ...] = () ``` ##### name `instance-attribute` ``` name: str ``` ##### unique `class-attribute` `instance-attribute` ``` unique: tuple[tuple[str, ...], ...] = () ``` Column sets that identify one row. ##### ddl ``` ddl() -> str ``` Return the table's `CREATE TABLE` statement. Returns: | Type | Description | | --- | --- | | `str` | The statement, with a trailing semicolon. | #### schema\_ddl ``` schema_ddl() -> str ``` Return the statements that create every table of the control schema. Returns: | Type | Description | | --- | --- | | `str` | DDL for :func:`timenet.format.duckdb.create_control_schema`. | ### control\_writer Write the in-memory TimeF hierarchy into `control.duckdb`. #### DuckDBControlWriter Serialize the relational part of one immutable TimeF version. ##### external `instance-attribute` ``` external = external ``` ##### path `instance-attribute` ``` path = Path(path) ``` ##### write\_hierarchy ``` write_hierarchy( dataset: TimeFDataset, placements: dict[tuple[int, int], ChunkPlacement] | None = None, *, tasks: Iterable[Task] | None = None, null_counts: Mapping[int, int] | None = None, ) -> dict[str, int] ``` Write records, sources, signals, axes, and their annotations atomically. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `dataset` | `TimeFDataset` | The complete in-memory hierarchy. | *required* | | `placements` | `dict[tuple[int, int], ChunkPlacement] | None` | Values-plane chunks keyed by `(signal_id, chunk_index)`. | `None` | | `tasks` | `Iterable[Task] | None` | A previously validated task stream, or `None` to use the dataset's tasks. | `None` | | `null_counts` | `Mapping[int, int] | None` | The number of missing timesteps for each signal id. Signals absent from this mapping use their own `null_count`. | `None` | Returns: | Type | Description | | --- | --- | | `dict[str, int]` | Counts keyed by concrete task type. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If the hierarchy refers to an object outside the dataset, repeats a task id, holds a value that JSON cannot store, or declares Signal lengths that do not match its axis offsets or value chunks. | ### duckdb DuckDB schema and connection helpers for the TimeF relational control plane. #### CATALOG\_ERRORS `module-attribute` ``` CATALOG_ERRORS = ( duckdb.CatalogException, duckdb.BinderException, duckdb.ConversionException, ) ``` DuckDB errors that mean a catalog lacks a table, column, or type that a read expects. #### CONTROL\_FILE `module-attribute` ``` CONTROL_FILE = 'control.duckdb' ``` Name of the relational control-plane database in a TimeF version. #### CONTROL\_SCHEMA\_VERSION `module-attribute` ``` CONTROL_SCHEMA_VERSION = 2 ``` Schema version written into :data:`CONTROL_FILE`, and the newest that this TimeNet reads. #### OLDEST\_READABLE\_CONTROL\_SCHEMA\_VERSION `module-attribute` ``` OLDEST_READABLE_CONTROL_SCHEMA_VERSION = 2 ``` The oldest schema version of :data:`CONTROL_FILE` that this TimeNet reads. Version 2 is the schema of TimeNet 0.12. Version 1 names layouts used during early development, and no TimeNet reads them. #### check\_control\_schema ``` check_control_schema(connection: DuckDBPyConnection) -> int ``` Return the version of a supported control database. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `connection` | `DuckDBPyConnection` | An open TimeF control database. | *required* | Raises: | Type | Description | | --- | --- | | `TimeFFormatVersionError` | If the schema is older or newer than this TimeNet reads. | | `TimeFFormatError` | If the schema metadata is missing or malformed. | #### connect\_control ``` connect_control( path: Path, *, read_only: bool = False ) -> DuckDBPyConnection ``` Open a TimeF control database. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `path` | `Path` | Local path to `control.duckdb`. | *required* | | `read_only` | `bool` | Open an immutable published database without write access. | `False` | Returns: | Type | Description | | --- | --- | | `DuckDBPyConnection` | An open DuckDB connection owned by the caller. | #### create\_control\_schema ``` create_control_schema( connection: DuckDBPyConnection, ) -> None ``` Create every control-plane table in one transaction. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `connection` | `DuckDBPyConnection` | A writable connection to a new database. | *required* | #### transaction ``` transaction( connection: DuckDBPyConnection, ) -> Generator[None] ``` Commit a block atomically or roll it back when any operation fails. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `connection` | `DuckDBPyConnection` | The connection that owns the transaction. | *required* | Yields: | Type | Description | | --- | --- | | `Generator[None]` | Control while the transaction is open. | ### schemas Pinned Arrow schema of the Parquet value shards. Shards hold no public ids. The DuckDB control plane maps each signal to its chunk placements. #### shard\_schema ``` shard_schema( value_type: DataType = _DEFAULT_VALUES_TYPE, ) -> Schema ``` Return the waveform shard schema. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `value_type` | `DataType` | The element type of the `values` list (defaults to `float32`). | `_DEFAULT_VALUES_TYPE` | Returns: | Type | Description | | --- | --- | | `Schema` | The Arrow schema. | ### storage Write-time collection of the payload sources a new release reads, pinned to their files. #### StorageSources Collect the releases that store a new release's reused payloads, and pin the files it reads. With a target registry, each payload source is checked there at its first reuse. The registry must hold the source with the same records of the reused files as the release the values were read from, and the source must not read payloads of the new release itself. ##### registry `instance-attribute` ``` registry = registry ``` ##### sources `property` ``` sources: dict[SourceKey, PayloadSource] ``` Each payload source with the files the new release reads from it, for the manifest. ##### target `instance-attribute` ``` target = target ``` ##### add ``` add( release: DatasetVersion, location: PayloadLocation, *, nullable: bool = False, ) -> SourceKey ``` Check a reused payload's files and pin them on the release that stores them. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `release` | `DatasetVersion` | The release whose catalog records `location`. The location's source names one of its payload sources, or is `None` when `release` stores the payload itself. | *required* | | `location` | `PayloadLocation` | Where the payload's bytes live. | *required* | | `nullable` | `bool` | Whether the spec of a signal array is nullable. Zarr stores the array's validity in parallel arrays, which the source must then list too. | `False` | Returns: | Type | Description | | --- | --- | | `SourceKey` | Its source key in the new release: the exact release as `org/name@version`. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If the storing release is the output release, its manifest lacks a file of the payload, or the target registry holds a source that reads payloads of the output release. | | `TimeNetDatasetNotFoundError` | If the target registry does not hold the source. | | `TimeFFormatError` | If the target registry's source records a reused file differently. | ### stored\_array Signal arrays stored as chunks in the Parquet or Zarr values files of a release. #### ChunkLocation `dataclass` Locate a signal array stored as chunks in the values files of one release. The catalog records each chunk's file, coordinates, and length, so the array's place in storage is known without opening a values file. ##### chunks `instance-attribute` ``` chunks: tuple[StoredChunk, ...] ``` The array's chunks in step order. ##### source `class-attribute` `instance-attribute` ``` source: SourceKey | None = None ``` The payload source that stores the chunks, or `None` for the release whose catalog records them. #### ChunkReader Bases: `Protocol` Find and read the stored chunks of one release's signal arrays. A release's `TimeFReader` is one: it finds the chunks of a batch of signals at a time, and reads them from whichever release stores them. #### StoredArray `dataclass` Bases: `ArrayHandle` Refer to one signal's array, stored as chunks in the values files of a release. The handle holds the release's reader and the signal's ID, spec, and number of values. A loaded handle also keeps the array and the location of its chunks in a write-once cell, so it reads, slices, and locates them without storage access. An unloaded handle holds no values: every read goes to the chunks, and `read_steps` reads only the chunks that cover the range. Handles from one reader for one signal are equal, whether loaded or not. ##### is\_loaded `property` ``` is_loaded: bool ``` Whether the handle keeps the values, so reads need no storage access. ##### n\_values `instance-attribute` ``` n_values: int ``` Number of values that the signal declares, which a read of every chunk must return. ##### reader `class-attribute` `instance-attribute` ``` reader: ChunkReader = dataclass_field(repr=False) ``` Reader of the release whose catalog records the signal. ##### signal\_id `instance-attribute` ``` signal_id: int ``` ID of the signal in that release's catalog. ##### spec `instance-attribute` ``` spec: TimeSeriesSpec ``` Spec that the stored values decode to. ##### read ``` read() -> Array ``` Return the kept values, or the values of every chunk. Returns: | Type | Description | | --- | --- | | `Array` | The signal's values. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the chunks hold another number of values than the signal declares. | ##### read\_numpy ``` read_numpy() -> ndarray ``` Return the values as a read-only NumPy array, shaped `(n_steps, *value_shape)`. The default converts :meth:`read`. A handle whose data is NumPy already returns it without building Arrow. Returns: | Type | Description | | --- | --- | | `ndarray` | The values, in the NumPy dtype of their Arrow type. | Raises: | Type | Description | | --- | --- | | `TimeFNullValuesError` | If the values contain nulls, which NumPy cannot hold. Read them with :meth:`read` instead. | ##### read\_steps ``` read_steps(start: int, stop: int) -> Array ``` Return the half-open step range `[start, stop)`. A loaded handle slices the values it keeps. An unloaded one reads only the chunks that cover the range. Returns: | Type | Description | | --- | --- | | `Array` | The steps' values, in the form :meth:`read` returns. | ### stored\_values Read and write text, string lists, and time offsets stored in Parquet payloads. #### ANNOTATION\_COLUMNS `module-attribute` ``` ANNOTATION_COLUMNS = StoredValueColumns( kind=ObjectKind.ANNOTATION_CONTENT, identity="content_id", fields={ PackedField.VALUE: { PackedValueKind.TEXT: "text_value", PackedValueKind.TEXT_LIST: "text_list_value", } }, ) ``` Packable columns of the annotation table. #### INLINE\_LIMIT `module-attribute` ``` INLINE_LIMIT = 4096 ``` The largest JSON-encoded text size, in bytes, that the writer keeps in the catalog. The writer stores larger text and lazy text handles in separate payload files to keep the catalog small. #### LOCATOR\_TABLE `module-attribute` ``` LOCATOR_TABLE = ModelTable[PackedValueLocator]( EXTERNAL_VALUES ) ``` #### PayloadLocation `module-attribute` ``` PayloadLocation = PackedLocation | ChunkLocation ``` Where a stored value's bytes live: packed text or time offsets, or a signal array's chunks. #### TARGET\_COLUMNS `module-attribute` ``` TARGET_COLUMNS = StoredValueColumns( kind=ObjectKind.TASK, identity="task_id", fields={ PackedField.TARGETS: { PackedValueKind.TEXT: "text_value" } }, position="position", ) ``` Packable columns of the task-target table. #### TASK\_COLUMNS `module-attribute` ``` TASK_COLUMNS = StoredValueColumns( kind=ObjectKind.TASK, identity="task_id", fields={ PackedField.PROMPT: { PackedValueKind.TEXT: "prompt" }, PackedField.RATIONALE: { PackedValueKind.TEXT: "rationale" }, }, ) ``` Packable columns of the task table. #### CatalogValue `dataclass` Hold a value kept in its catalog row, including a legitimate absent value. ##### value `instance-attribute` ``` value: object ``` #### ExternalObjectFieldKey `dataclass` Identify an external field on an object, including its position in an ordered field. The object is named the way the catalog names it: a task by its public id, an annotation's content by the storage id of its `annotation_contents` row, and an irregular axis by the storage id of its `axes` row. ##### field `instance-attribute` ``` field: PackedField ``` ##### object\_id `instance-attribute` ``` object_id: int ``` ##### object\_kind `instance-attribute` ``` object_kind: ObjectKind ``` ##### position `class-attribute` `instance-attribute` ``` position: int = 0 ``` #### ExternalValue `dataclass` Identify a value written to a payload instead of the catalog. ##### locator `instance-attribute` ``` locator: PackedValueLocator ``` #### PackedContentKey `dataclass` Name one packed value by its payload file's checksum and its place in the file, without a read. The checksum identifies the file's bytes, so equal keys mean equal values, and readers of one release, or of releases that share a payload file, give the same key. The key compares equal only to another `PackedContentKey`, so it never equals the key of text in memory, which is a string or a tuple of strings. ##### checksum `instance-attribute` ``` checksum: str ``` The payload file's checksum, as the manifest records it. ##### kind `instance-attribute` ``` kind: PackedValueKind ``` Kind of the value, which names the payload column that holds it. ##### row `instance-attribute` ``` row: int ``` Row of the value in the payload file. #### PackedField Bases: `StrEnum` Name the object fields that can store external values. ##### PROMPT `class-attribute` `instance-attribute` ``` PROMPT = 'prompt' ``` ##### RATIONALE `class-attribute` `instance-attribute` ``` RATIONALE = 'rationale' ``` ##### TARGETS `class-attribute` `instance-attribute` ``` TARGETS = 'targets' ``` ##### TIME\_OFFSETS `class-attribute` `instance-attribute` ``` TIME_OFFSETS = 'time_offsets' ``` ##### VALUE `class-attribute` `instance-attribute` ``` VALUE = 'value' ``` #### PackedLocation `dataclass` Locate one value packed in a Parquet payload file of a release. The catalog records every field, so the value's release, kind, length, and digest are known without opening the payload file. ##### digest `class-attribute` `instance-attribute` ``` digest: bytes | None = None ``` SHA-256 digest of time offsets, which identifies them without a read. `None` for text. ##### kind `instance-attribute` ``` kind: PackedValueKind ``` Kind of the value, which names the payload column that holds it. ##### length `instance-attribute` ``` length: int ``` Number of code points in a string, of items in a string list, or of time offsets. ##### path `instance-attribute` ``` path: str ``` Payload file, relative to the root of the release that stores it. ##### row `instance-attribute` ``` row: int ``` Row of the value in the payload file. ##### source `class-attribute` `instance-attribute` ``` source: SourceKey | None = None ``` The payload source that stores the file, or `None` for the release whose catalog records it. #### PackedValueKind Bases: `StrEnum` Represent the value shape stored at a packed payload location. ##### TEXT `class-attribute` `instance-attribute` ``` TEXT = 'text' ``` ##### TEXT\_LIST `class-attribute` `instance-attribute` ``` TEXT_LIST = 'texts' ``` ##### TIME\_OFFSETS `class-attribute` `instance-attribute` ``` TIME_OFFSETS = 'time_offsets' ``` #### PackedValueLocator Bases: `TimeFModel` Validate the catalog location and value kind of one external field. ##### field `instance-attribute` ``` field: PackedField ``` ##### key `property` ``` key: ExternalObjectFieldKey ``` The validated object field addressed by this locator. ##### location `property` ``` location: PackedLocation ``` Where the value is packed. ##### model\_config `class-attribute` `instance-attribute` ``` model_config = ConfigDict( extra="forbid", frozen=True, revalidate_instances="always", serialize_by_alias=True, validate_by_name=True, validate_default=True, ) ``` ##### n\_values `class-attribute` `instance-attribute` ``` n_values: int = Field(strict=True, ge=0) ``` ##### object\_id `class-attribute` `instance-attribute` ``` object_id: int = Field(strict=True, gt=0) ``` ##### object\_kind `instance-attribute` ``` object_kind: ObjectKind ``` ##### position `class-attribute` `instance-attribute` ``` position: int = Field(strict=True, ge=0) ``` ##### row\_index `class-attribute` `instance-attribute` ``` row_index: int = Field(strict=True, ge=0) ``` ##### source\_key `class-attribute` `instance-attribute` ``` source_key: SourceKey | None = Field( default=None, strict=True, min_length=1 ) ``` ##### value\_digest `class-attribute` `instance-attribute` ``` value_digest: str | None = Field( default=None, pattern=f"^{CHECKSUM_PREFIX}[0-9a-f]{64}$", strict=True, ) ``` ##### value\_kind `instance-attribute` ``` value_kind: PackedValueKind ``` ##### value\_path `class-attribute` `instance-attribute` ``` value_path: str = Field(strict=True, min_length=1) ``` #### PackedValueReader Read one release's packed values through a bounded cache of decoded payload files. A value whose location names a source is read from that payload source of the release. ##### version `instance-attribute` ``` version = version ``` ##### checksum ``` checksum(location: PackedLocation) -> str ``` Return the checksum that the release storing the location's payload file records for it. The location's source names that release: one of this release's payload sources, or this release itself for `None`. Its manifest lists the file, so the checksum needs no read. Returns: | Type | Description | | --- | --- | | `str` | The file's checksum, with its `sha256:` prefix. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the source is unknown, or its manifest does not list the file. | ##### close ``` close() -> None ``` Release cached payload tables. ##### content\_key ``` content_key(location: PackedLocation) -> PackedContentKey ``` Return the key that names a packed value by its payload file's checksum, without a read. Returns: | Type | Description | | --- | --- | | `PackedContentKey` | The checksum the manifest records for the location's file, with the row and kind. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the manifest does not list the location's payload file. | ##### handle ``` handle( location: PackedLocation, ) -> StoredText | StoredTextList ``` Return a text handle of the location's value kind, without reading the payload. Returns: | Type | Description | | --- | --- | | `StoredText | StoredTextList` | A `StoredText` for a string, or a `StoredTextList` for a list of strings. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the manifest does not list the location's payload file. | | `TimeFValidationError` | If the location holds time offsets, whose `StoredOffsets` also needs their axis. | ##### handles ``` handles( connection: DuckDBPyConnection, kind: ObjectKind, object_ids: Iterable[int], ) -> dict[ ExternalObjectFieldKey, StoredText | StoredTextList ] ``` Return text handles for one batch of objects' packed values without opening payload files. Returns: | Type | Description | | --- | --- | | `dict[ExternalObjectFieldKey, StoredText | StoredTextList]` | Handles keyed by object kind, object id, field, and position. | ##### locations `staticmethod` ``` locations( connection: DuckDBPyConnection, kind: ObjectKind, object_ids: Iterable[int], ) -> dict[ExternalObjectFieldKey, PackedLocation] ``` Return where one batch of objects' values are packed, without opening payload files. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `connection` | `DuckDBPyConnection` | The release's open control database. | *required* | | `kind` | `ObjectKind` | The kind of the objects that own the values. | *required* | | `object_ids` | `Iterable[int]` | The catalog ids of the objects, such as task ids or annotation content ids. | *required* | Returns: | Type | Description | | --- | --- | | `dict[ExternalObjectFieldKey, PackedLocation]` | Locations keyed by object kind, object id, field, and position. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the catalog has no locator table, or a locator is invalid or duplicated. | ##### offsets ``` offsets( location: PackedLocation, *, axis_id: int, first: int, last: int, ) -> StoredOffsets ``` Return a handle to one irregular axis's packed time offsets, without reading the payload. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `location` | `PackedLocation` | Where the time offsets are packed. | *required* | | `axis_id` | `int` | The axis's catalog ID, for error messages. | *required* | | `first` | `int` | The first time offset the axis declares, in microseconds. | *required* | | `last` | `int` | The last time offset the axis declares, in microseconds. | *required* | Returns: | Type | Description | | --- | --- | | `StoredOffsets` | A `StoredOffsets` that checks the time offsets against the axis on its first read. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the manifest does not list the location's payload file. | ##### read\_scalar ``` read_scalar(location: PackedLocation) -> Scalar ``` Read a packed value from the payload file of the release that stores it. Returns: | Type | Description | | --- | --- | | `Scalar` | The Arrow scalar at the location's file and row: a string, or a list of strings or time offsets. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the payload file's schema, the location, or the stored value is invalid. | #### PackedValueWriter Pack new values into Parquet payload files, and reuse the payloads that store read values. ##### bytes `instance-attribute` ``` bytes = 0 ``` ##### files `instance-attribute` ``` files: list[str] = [] ``` ##### locators `instance-attribute` ``` locators: list[PackedValueLocator] = [] ``` ##### root `instance-attribute` ``` root = root ``` ##### sources `instance-attribute` ``` sources = sources ``` ##### text\_lists `instance-attribute` ``` text_lists: list[list[str] | None] = [] ``` ##### texts `instance-attribute` ``` texts: list[str | None] = [] ``` ##### time\_offsets `instance-attribute` ``` time_offsets: list[Array | None] = [] ``` ##### finish ``` finish(connection: DuckDBPyConnection) -> None ``` Flush payload files and store the remaining catalog locators. ##### flush ``` flush() -> None ``` Write one packed group and release its buffered values. ##### pack ``` pack( kind: ObjectKind, object_id: int, field: PackedField, value: object, position: int = 0, *, digest: bytes | None = None, ) -> CatalogValue | ExternalValue ``` Store a value in the catalog or a separate payload file. Time offsets, :class:`LazyText`, and :class:`LazyTextList` go into payload files. If other text exceeds :data:`INLINE_LIMIT`, it also goes into a payload file. Stored handles reuse their existing payload rows. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `kind` | `ObjectKind` | The kind of object that owns the value. | *required* | | `object_id` | `int` | The object id in the output catalog: a task id, an annotation content id, or an axis id. | *required* | | `field` | `PackedField` | The field that holds the value. | *required* | | `value` | `object` | The text, time offsets, or other catalog value to store. | *required* | | `position` | `int` | The position in a field with several values, such as task targets. | `0` | | `digest` | `bytes | None` | An optional `offsets_digest()` result for time offsets. If omitted, this method calculates it. | `None` | Returns: | Type | Description | | --- | --- | | `CatalogValue | ExternalValue` | A catalog value or a locator that identifies the value in a payload file. | Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If the locator is invalid for this field, the output release stores a reused payload, or the source release lacks a reused payload. | ##### write\_locators ``` write_locators( connection: DuckDBPyConnection, *, at_least: int = 0 ) -> None ``` Store the locators packed so far in the catalog and release them. The catalog writer calls this once per batch inside its transaction, so a streamed source never holds all of its locators. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `connection` | `DuckDBPyConnection` | The open write transaction. | *required* | | `at_least` | `int` | Keep the locators while there are fewer than this many. | `0` | #### StoredOffsets `dataclass` Bases: `ArrayHandle` Refer to the time offsets of one irregular axis, packed in a Parquet payload. The handle holds the reader of the release's payloads, the time offsets' location, and the axis's ID and endpoints. Its first `read()` checks the time offsets against the location and the axis and keeps a compact copy in a write-once cell, so from then on the handle is loaded and reads need no payload access. The location records the number of time offsets and their digest, so both are known without a read. ##### axis\_id `instance-attribute` ``` axis_id: int ``` Catalog ID of the axis the time offsets belong to. ##### digest `property` ``` digest: bytes ``` The time offsets' SHA-256 digest, recorded in their location. The catalog stores a digest with every axis's time offsets, so only a location built by hand can lack one. Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the location records no digest. | ##### first `instance-attribute` ``` first: int ``` First time offset the axis declares, in microseconds. ##### is\_loaded `property` ``` is_loaded: bool ``` Whether a read has kept the time offsets. ##### last `instance-attribute` ``` last: int ``` Last time offset the axis declares, in microseconds. ##### length `property` ``` length: int ``` The number of time offsets, recorded in their location. ##### store `class-attribute` `instance-attribute` ``` store: PackedValueReader = dataclass_field(repr=False) ``` ##### read ``` read() -> Array ``` Return the time offsets, read and checked on the first call and kept. Returns: | Type | Description | | --- | --- | | `Array` | One int64 microsecond offset per step. | ##### read\_numpy ``` read_numpy() -> ndarray ``` Return the values as a read-only NumPy array, shaped `(n_steps, *value_shape)`. The default converts :meth:`read`. A handle whose data is NumPy already returns it without building Arrow. Returns: | Type | Description | | --- | --- | | `ndarray` | The values, in the NumPy dtype of their Arrow type. | Raises: | Type | Description | | --- | --- | | `TimeFNullValuesError` | If the values contain nulls, which NumPy cannot hold. Read them with :meth:`read` instead. | ##### read\_steps ``` read_steps(start: int, stop: int) -> Array ``` Return the half-open step range `[start, stop)`. The default slices :meth:`read`, so a `stop` past the end reads to the end. A stored handle reads only the chunks that cover the range. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `start` | `int` | First step, inclusive, and not negative. | *required* | | `stop` | `int` | Last step, exclusive, and not before `start`. | *required* | Returns: | Type | Description | | --- | --- | | `Array` | The steps' values, in the form :meth:`read` returns. | #### StoredText `dataclass` Bases: `TextHandle` Refer to one string packed in a Parquet payload. The handle holds the reader of the release's payloads and the string's location. A loaded handle also keeps the string in a write-once cell, and `read()` returns it without payload access. An unloaded handle holds no text, and every `read()` reads the payload. Handles at one location of one payload file are equal without a read, whether loaded or not. ##### is\_loaded `property` ``` is_loaded: bool ``` Whether the handle keeps the string, so reads need no payload access. ##### length `property` ``` length: int ``` The string's length, recorded in its location. ##### store `class-attribute` `instance-attribute` ``` store: PackedValueReader = dataclass_field(repr=False) ``` ##### read ``` read() -> str ``` Return the kept string, or the string from its payload. #### StoredTextList `dataclass` Bases: `TextListHandle` Refer to a list of strings packed in a Parquet payload. The handle holds the reader of the release's payloads and the list's location. A loaded handle also keeps the strings in a write-once cell, and `read()` returns a new list of them without payload access. An unloaded handle holds no text, and every `read()` reads the payload. Handles at one location of one payload file are equal without a read, whether loaded or not. ##### is\_loaded `property` ``` is_loaded: bool ``` Whether the handle keeps the strings, so reads need no payload access. ##### length `property` ``` length: int ``` The number of strings, recorded in the list's location. ##### store `class-attribute` `instance-attribute` ``` store: PackedValueReader = dataclass_field(repr=False) ``` ##### read ``` read() -> list[str] ``` Return the kept strings, or the list of strings from its payload, as a new list. #### StoredValueColumns `dataclass` Name the columns of one public Arrow table whose values can be packed. ##### fields `instance-attribute` ``` fields: Mapping[PackedField, Mapping[PackedValueKind, str]] ``` Per packed field, the column that holds each value kind. ##### identity `instance-attribute` ``` identity: str ``` Column holding each row's owner id: a task id, or an annotation content id. ##### kind `instance-attribute` ``` kind: ObjectKind ``` Object kind whose locators address the packed values. ##### position `class-attribute` `instance-attribute` ``` position: str | None = None ``` Column holding the ordered position, for a field with several values per object. #### materialize\_table ``` materialize_table( table: Table, reader: PackedValueReader, connection: DuckDBPyConnection, columns: StoredValueColumns, ) -> Table ``` Resolve external text fields while retaining the public Arrow table schema. Returns: | Type | Description | | --- | --- | | `Table` | The original table when it has no external values, otherwise a table with native values. | #### offsets\_digest ``` offsets_digest(offsets: ndarray | Array) -> bytes ``` Return the SHA-256 digest of an irregular axis's time offsets. Every caller hashes the same bytes, contiguous little-endian int64, so one sequence of time offsets has one digest whether it is held as NumPy or as Arrow. Returns: | Type | Description | | --- | --- | | `bytes` | The 32-byte digest. | ### table\_writer Insert catalog rows using their shared table definitions. #### ModelTable `dataclass` Bases: `Generic[_Row]` Bind validated model rows to one catalog table definition. ##### definition `instance-attribute` ``` definition: Table ``` ##### write ``` write( connection: DuckDBPyConnection, rows: Sequence[_Row] ) -> None ``` Serialize model rows and insert them through the shared catalog writer. #### insert\_rows ``` insert_rows( connection: DuckDBPyConnection, table: Table, rows: Sequence[Mapping[str, object]], ) -> None ``` Insert one Arrow batch using the catalog table's schema and column order. Raises: | Type | Description | | --- | --- | | `TimeFValidationError` | If the rows cannot be converted or inserted into the table. | ### task\_codec Canonical codecs between Task fields and the typed columns of the `tasks` and `task_targets` tables. Spans, task configuration, and targets are all stored as plain columns rather than JSON, so DuckDB can filter on them and the reader builds objects straight from row values. Object references are public IDs. The target rows use the `task_targets` column names, so the rows are stored and read back unchanged. A span holds Signal objects, so encoding reads their ids and decoding looks the read-back Signals up by id. #### ANNOTATION\_FIELDS `module-attribute` ``` ANNOTATION_FIELDS = frozenset( { TaskRelationshipField.INPUT_ANNOTATIONS, TaskRelationshipField.TARGET_ANNOTATIONS, } ) ``` The fields stored in `task_annotation_refs`. #### PAYLOAD\_COLUMNS `module-attribute` ``` PAYLOAD_COLUMNS = ( "label_set", "unit", "target_name", "mode", ) ``` The `tasks` columns that hold configuration for each task type. Unused columns are NULL. A classification task stores its label set name in `label_set`. The `label_sets` table stores the set itself. #### RECORD\_FIELDS `module-attribute` ``` RECORD_FIELDS = frozenset( { TaskRelationshipField.INPUTS, TaskRelationshipField.CANDIDATE_RECORDS, } ) ``` The fields stored in `task_record_refs`. #### TaskRelationshipField Bases: `StrEnum` Task field that a `task_record_refs` or `task_annotation_refs` row belongs to. ##### CANDIDATE\_RECORDS `class-attribute` `instance-attribute` ``` CANDIDATE_RECORDS = 'candidate_records' ``` ##### INPUTS `class-attribute` `instance-attribute` ``` INPUTS = 'inputs' ``` ##### INPUT\_ANNOTATIONS `class-attribute` `instance-attribute` ``` INPUT_ANNOTATIONS = 'input_annotations' ``` ##### TARGET\_ANNOTATIONS `class-attribute` `instance-attribute` ``` TARGET_ANNOTATIONS = 'target_annotations' ``` #### decode\_input\_modalities ``` decode_input_modalities( values: list[str] | None, ) -> frozenset[InputModality] ``` Decode a required stored input modality set. Returns: | Type | Description | | --- | --- | | `frozenset[InputModality]` | The declared input kinds. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If a stored input kind is unknown or repeated. | #### decode\_span ``` decode_span( span_type: str | None, span_start: int | None, span_end: int | None, signal_ids: Iterable[EntityId] | None, *, signals: Mapping[EntityId, Signal], ) -> Span | None ``` Decode the four span columns. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `span_type` | `str | None` | The stored span shape, or `None` for no span. | *required* | | `span_start` | `int | None` | The stored start bound. | *required* | | `span_end` | `int | None` | The stored end bound. | *required* | | `signal_ids` | `Iterable[EntityId] | None` | The ids of the Signals the span is scoped to. | *required* | | `signals` | `Mapping[EntityId, Signal]` | The read-back Signals by id, which take the place of `signal_ids`. | *required* | Returns: | Type | Description | | --- | --- | | `Span | None` | The decoded Span, or `None` when `span_type` is NULL. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the stored span type is unknown, a required bound is NULL, or a Signal id names no read-back Signal. | #### decode\_split ``` decode_split(value: str | None) -> Split | None ``` Decode a stored split. Returns: | Type | Description | | --- | --- | | `Split | None` | The task's partition, or `None` for a task its dataset does not partition. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the stored split is unknown. | #### decode\_target ``` decode_target( row: Mapping[str, object] | TargetValueRow, *, records: Mapping[EntityId, Record], signals: Mapping[EntityId, Signal], ) -> object ``` Decode one typed target row and resolve stored object references. Returns: | Type | Description | | --- | --- | | `object` | The native scalar, Record, Signal, or Span target. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If a required value or referenced object is missing, or the kind is unknown. | #### decode\_task\_payload ``` decode_task_payload( task_type: TaskType, columns: dict[str, str | None] ) -> dict[str, Any] ``` Decode concrete Task configuration from the payload columns. Returns: | Type | Description | | --- | --- | | `dict[str, Any]` | Keyword arguments for the concrete Task constructor. | #### encode\_span ``` encode_span(span: Span | None) -> dict[str, Any] | None ``` Encode a concrete span as the four span columns without losing its frame or shape. Returns: | Type | Description | | --- | --- | | `dict[str, Any] | None` | `{"span_type", "span_start", "span_end", "signal_ids"}`, or `None`. `signal_ids` is the | | `dict[str, Any] | None` | span's series scope: `None` for every series, a tuple of IDs for a time span, and a | | `dict[str, Any] | None` | one-element tuple for a step span. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the Span subtype is unsupported. | #### encode\_task\_payload ``` encode_task_payload(task: Task) -> dict[str, str | None] ``` Encode the configuration owned by the concrete Task class as the payload columns. Returns: | Type | Description | | --- | --- | | `dict[str, str | None]` | One value per :data:`PAYLOAD_COLUMNS`, `None` where the task type has no such field. | ### task\_target\_kind Tags for the scalar, object, and span targets stored in the control catalog. #### SPAN\_KINDS `module-attribute` ``` SPAN_KINDS = frozenset( { TaskTargetKind.TIME_POINT, TaskTargetKind.TIME_INTERVAL, TaskTargetKind.STEP_POINT, TaskTargetKind.STEP_INTERVAL, } ) ``` The kinds that store a span. `tasks.scope_type` holds one of these too. #### TaskTargetKind Bases: `StrEnum` Identify the value columns used by one stored task target. ##### BOOLEAN `class-attribute` `instance-attribute` ``` BOOLEAN = 'boolean' ``` ##### FLOAT `class-attribute` `instance-attribute` ``` FLOAT = 'float' ``` ##### INTEGER `class-attribute` `instance-attribute` ``` INTEGER = 'integer' ``` ##### RECORD `class-attribute` `instance-attribute` ``` RECORD = 'record' ``` ##### SIGNAL `class-attribute` `instance-attribute` ``` SIGNAL = 'signal' ``` ##### STEP\_INTERVAL `class-attribute` `instance-attribute` ``` STEP_INTERVAL = 'step_interval' ``` ##### STEP\_POINT `class-attribute` `instance-attribute` ``` STEP_POINT = 'step_point' ``` ##### TEXT `class-attribute` `instance-attribute` ``` TEXT = 'text' ``` ##### TIME\_INTERVAL `class-attribute` `instance-attribute` ``` TIME_INTERVAL = 'time_interval' ``` ##### TIME\_POINT `class-attribute` `instance-attribute` ``` TIME_POINT = 'time_point' ``` #### is\_step\_span ``` is_step_span(kind: str) -> bool ``` Return whether a stored target kind uses a single step-indexed Signal. ### task\_target\_row Validated target values and catalog rows. #### TargetValueRow Bases: `TimeFModel` Represent a public target with validated values and public object IDs. ##### boolean\_value `class-attribute` `instance-attribute` ``` boolean_value: StrictBool | None = None ``` ##### float\_value `class-attribute` `instance-attribute` ``` float_value: StrictFloat | None = None ``` ##### integer\_value `class-attribute` `instance-attribute` ``` integer_value: StrictInt | None = None ``` ##### model\_config `class-attribute` `instance-attribute` ``` model_config = ConfigDict( extra="forbid", frozen=True, revalidate_instances="always", serialize_by_alias=True, validate_by_name=True, validate_default=True, ) ``` ##### record\_id `class-attribute` `instance-attribute` ``` record_id: StrictInt | None = None ``` ##### signal\_id `class-attribute` `instance-attribute` ``` signal_id: StrictInt | None = None ``` ##### span\_end `class-attribute` `instance-attribute` ``` span_end: StrictInt | None = None ``` ##### span\_signal\_ids `class-attribute` `instance-attribute` ``` span_signal_ids: tuple[StrictInt, ...] | None = None ``` ##### span\_start `class-attribute` `instance-attribute` ``` span_start: StrictInt | None = None ``` ##### target\_kind `instance-attribute` ``` target_kind: TaskTargetKind ``` ##### text\_value `class-attribute` `instance-attribute` ``` text_value: StrictStr | None = None ``` ##### from\_target `classmethod` ``` from_target(target: object) -> Self ``` Construct a validated row for one public target. Returns: | Type | Description | | --- | --- | | `Self` | The target's value columns, with unrelated columns left NULL. | Raises: | Type | Description | | --- | --- | | `TimeFFormatError` | If the target type is unsupported. |