# Timenet.reader

## timenet.reader

Deserialization of a TimeF version directory into a :class:`~timenet.dataset.TimeFDataset`.

### RECORD\_BATCH\_SIZE `module-attribute`

```
RECORD_BATCH_SIZE = 1000
```

The most Records one batch of a Record stream holds, unless the stream sets another batch size.

### TASK\_BATCH\_SIZE `module-attribute`

```
TASK_BATCH_SIZE = 5000
```

The most tasks one batch of a task read or stream holds, unless a stream sets another batch size.

### LoadPlan `dataclass`

Specify, for each kind of stored value, whether a read loads it or leaves it lazy.

A read applies its plan to the objects it builds: a dataset's records, tasks, and annotations; a
record's signals, time offsets, and annotations; a task's text, annotations, and the records,
signals, and annotations it references. It loads nothing on objects passed to it, such as the
records given to `read_tasks`, and nothing on tasks it builds only as another task's
`from_tasks`. A plan never widens what a read reaches.

#### ALL\_EAGER `class-attribute`

```
ALL_EAGER: LoadPlan
```

Load every kind, so the objects a read builds need no storage afterwards.

#### ALL\_LAZY `class-attribute`

```
ALL_LAZY: LoadPlan
```

Load nothing. Every read's default.

#### signal\_values `class-attribute` `instance-attribute`

```
signal_values: Loading = Loading.LAZY
```

Signal arrays.

#### text `class-attribute` `instance-attribute`

```
text: Loading = Loading.LAZY
```

Packed text: prompts, rationales, text targets, and annotation strings and string lists.

#### time\_offsets `class-attribute` `instance-attribute`

```
time_offsets: Loading = Loading.LAZY
```

Time offsets of irregular axes, one array per axis that its signals share.

### Loading

Bases: `StrEnum`

How a read delivers one kind of stored value on the objects it builds.

#### EAGER `class-attribute` `instance-attribute`

```
EAGER = 'eager'
```

Loaded during the read and kept in memory in its value handle, so later reads don't touch storage.

#### LAZY `class-attribute` `instance-attribute`

```
LAZY = 'lazy'
```

Read from storage when accessed.

### TimeFReader

Read a committed TimeF dataset through its DuckDB control plane.

#### metadata `property`

```
metadata: DatasetMetadata
```

The dataset's descriptive metadata.

#### schema `property`

```
schema: DatasetSchema
```

The dataset's type declaration.

#### values\_backend `property`

```
values_backend: str
```

The values-plane backend named by the manifest.

#### close

```
close() -> None
```

Close cached handles while keeping the reader reusable.

#### get

```
get(
    kind: type[E],
    id: EntityId,
    *,
    parent: ParentAlias | DatasetRef | None = None,
    load: LoadPlan = ALL_LAZY,
) -> E
```

Read one object by its ID, from this dataset or imported from a parent.

An ID is an opaque handle. It stays the same across edits of one dataset, and it means
something only together with that dataset's reference: an object imported from a parent is
found by the parent's ID, with `parent` naming the parent. An ID is not a lookup key for
content; to find an object by what it is, read objects with :meth:`iter_records` or
:meth:`read_tasks` and check their `metadata`.

`get(Task, 7)` reads the task with ID 7; `get(AnswerTask, 7)` also checks that it is an
answer task. With `parent`, `id` is the ID the parent dataset gives the object, and the
read returns this dataset's imported copy.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `kind` | `type[E]` | `Record`, `Signal`, or a task class such as `Task` or `AnswerTask`. Sources and annotations are read through the objects that hold them. | *required* |
| `id` | `EntityId` | The object's ID, a positive int64. | *required* |
| `parent` | `ParentAlias | DatasetRef | None` | The parent the object was imported from: the `ParentAlias` it was imported under, or its exact release as a `DatasetRef`. | `None` |
| `load` | `LoadPlan` | Which kinds of stored values to load into the object and what it references. The default loads none. | `ALL_LAZY` |

Returns:

| Type | Description |
| --- | --- |
| `E` | The object. A Signal is part of its Record, which the read builds too. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFNotFoundError` | If the ID or `parent` names nothing, or the stored object is not a `kind`. |
| `TimeFValidationError` | If `kind` is `Source`, `Annotation`, or another kind not listed above, `id` is not a positive int64, such as a string, or `parent` is neither a `ParentAlias` nor a `DatasetRef`, such as a plain string. |
| `TimeFFormatError` | If the catalog names no parent release or another one for the alias than the manifest does, or lists the alias or a parent ID more than once. |

#### get\_many

```
get_many(
    kind: type[E],
    ids: Iterable[EntityId],
    *,
    parent: ParentAlias | DatasetRef | None = None,
    load: LoadPlan = ALL_LAZY,
) -> tuple[E, ...]
```

Read objects of one kind by their IDs, in one batch.

IDs are opaque handles, as :meth:`get` describes: stable across edits of this dataset,
meaningful only with its reference, and no key for content. The objects form one graph: a
Record that two of them reference is one object, and the Signals on one irregular axis
share one time-offsets handle. `load` loads their values in one batch.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `kind` | `type[E]` | `Record`, `Signal`, or a task class such as `Task` or `AnswerTask`. Sources and annotations are read through the objects that hold them. | *required* |
| `ids` | `Iterable[EntityId]` | The objects' IDs, positive int64s in a list, tuple, or other iterable. With `parent`, the IDs the parent dataset gives the objects. | *required* |
| `parent` | `ParentAlias | DatasetRef | None` | The parent the objects were imported from: the `ParentAlias` they were imported under, or its exact release as a `DatasetRef`. | `None` |
| `load` | `LoadPlan` | Which kinds of stored values to load into the objects and what they reference. The default loads none. | `ALL_LAZY` |

Returns:

| Type | Description |
| --- | --- |
| `tuple[E, ...]` | One object per ID, in the order of `ids`. A repeated ID returns the same object. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFNotFoundError` | If an ID or `parent` names nothing, or a stored object is not a `kind`. Nothing is returned then. |
| `TimeFValidationError` | If `kind` is `Source`, `Annotation`, or another kind not listed above, `ids` is a string or holds anything but positive int64s, or `parent` is neither a `ParentAlias` nor a `DatasetRef`, such as a plain string. |
| `TimeFFormatError` | If the catalog names no parent release or another one for the alias than the manifest does, or lists the alias or a parent ID more than once. |

#### iter\_records

```
iter_records(
    record_ids: Iterable[EntityId] | None = None,
    *,
    with_annotations: bool = True,
    load: LoadPlan = ALL_LAZY,
    batch_size: int = RECORD_BATCH_SIZE,
) -> Iterator[Record]
```

Yield complete records a bounded batch at a time, in a requested ID order or in record order.

The stream builds and loads each batch on its own, before it yields the batch. The records
of a batch share the objects that the batch builds. So the signals on one irregular axis
share one handle to the time offsets of that axis.

Batches share nothing. So after the stream yields the objects of a batch, it keeps none of
them. It holds one batch of objects at a time. If two batches request one record ID, each
batch builds that record as a separate object. The stream checks every requested ID before
it builds the first batch.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `record_ids` | `Iterable[EntityId] | None` | IDs to read in result order, or `None` for every record in the order of :meth:`record_ids`. | `None` |
| `with_annotations` | `bool` | Whether to build annotations on the records and descendants. | `True` |
| `load` | `LoadPlan` | The kinds of stored values to load into the records that each batch builds. The default loads none. | `ALL_LAZY` |
| `batch_size` | `int` | The most records that a batch holds. By default, a batch holds up to 1,000 records (`RECORD_BATCH_SIZE`). The stream holds one batch at a time, with every value that the plan loaded. So with an eager plan, a smaller batch holds fewer arrays in memory. | `RECORD_BATCH_SIZE` |

Yields:

| Type | Description |
| --- | --- |
| `Record` | Records with loaded values for each kind of value that the plan names. Other stored |
| `Record` | values stay behind their value handles. |

#### iter\_tasks

```
iter_tasks(
    *,
    inputs: Iterable[Record] | None = None,
    split: Split | str | None = None,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    load: LoadPlan = ALL_LAZY,
    batch_size: int = TASK_BATCH_SIZE,
) -> Iterator[Task]
```

Yield the tasks :meth:`read_tasks` reads, a bounded batch at a time.

Each batch is built and loaded on its own. Its tasks share the objects it builds: a record
that two of them use is one object, and the signals on one irregular axis share one
time-offsets handle. Batches share nothing but `inputs`, so the stream keeps none of a
batch's objects once it has yielded them: it holds one batch of objects and that batch's
task IDs at a time, and selects the next batch's IDs after the last one, while the reader
keeps compact chunk locators for the signals it has resolved until it closes. A record or
parent task that tasks of several batches use is built once per batch, as separate objects
with the same ID, each loaded as far as its own batch's plan reaches it.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `inputs` | `Iterable[Record] | None` | Keep only tasks that use one of these Records as input, or `None` for every task. Tasks reference these `Record` objects directly, and the read leaves them as they are. | `None` |
| `split` | `Split | str | None` | Keep only tasks of this split, or `None` for every task. The filter runs in DuckDB, so a task outside the split is never built. | `None` |
| `required_modalities` | `Iterable[InputModality] | None` | Keep only tasks that declare every one of these input modalities. | `None` |
| `supported_modalities` | `Iterable[InputModality] | None` | Keep only tasks whose declared input modalities are all among these. | `None` |
| `load` | `LoadPlan` | Which kinds of stored values to load into the objects each batch builds. The default loads none. A batch builds but does not load what it reaches only through its tasks' `from_tasks` parents, which is the parents and their records and annotations; a record or annotation that a task of the batch uses as well loads. | `ALL_LAZY` |
| `batch_size` | `int` | The most tasks a batch holds. By default, a batch holds up to 5,000 tasks (`TASK_BATCH_SIZE`). Pass another value for larger or smaller batches. The stream holds one batch at a time, with every value its plan loaded, so with an eager plan a smaller batch holds fewer arrays in memory. | `TASK_BATCH_SIZE` |

Yields:

| Type | Description |
| --- | --- |
| `Task` | Concrete tasks in stable ID order. |

#### open `classmethod`

```
open(
    path: str | Path,
    *,
    sources_root: str | Path | None = None,
) -> TimeFReader
```

Open a local version directory for reading.

Opening reads only metadata, as :func:`~timenet.registry.dependencies.open_local_version`
does. The reader opens the catalog on its first read. Close the reader after use, or use it
as a context manager.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `path` | `str | Path` | The version directory, which holds `manifest.json` and `control.duckdb`. | *required* |
| `sources_root` | `str | Path | None` | The registry root to find payload sources under. By default this is the root that holds `path` as `<root>/<org>/<name>/<version>`. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `TimeFReader` | A reader for the version. |

Raises:

| Type | Description |
| --- | --- |
| `FileNotFoundError` | If `path` has no manifest. |
| `TimeFValidationError` | If the version has payload sources, is not stored as `<root>/<org>/<name>/<version>`, and no `sources_root` is given. |
| `TimeNetMissingParentError` | If a payload source is not under the sources root. |

#### read

```
read(*, load: LoadPlan = ALL_LAZY) -> TimeFDataset
```

Read the whole dataset, loading the kinds of stored values that `load` names.

Every call builds new objects. With the default, every stored value stays behind its value
handle and reads through to storage on access. With `LoadPlan.ALL_EAGER`, the dataset
needs no storage after the reader closes.

The dataset gets the parent cards and the lineage of the manifest, and the import map of the
catalog, so a later import into it reuses the copies it holds instead of copying them again.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `load` | `LoadPlan` | Which kinds of stored values to load into the objects the read builds. | `ALL_LAZY` |

Returns:

| Type | Description |
| --- | --- |
| `TimeFDataset` | The dataset. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFFormatError` | If the catalog's parent aliases or their parent releases differ from the manifest's parent cards, or the catalog lists an alias or a parent ID more than once. |

#### read\_annotation\_table

```
read_annotation_table(
    object_type: ObjectKind | None = None,
) -> Table
```

Return every annotation occurrence with its content as an Arrow table.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `object_type` | `ObjectKind | None` | Keep occurrences on this object kind, such as `ObjectKind.RECORD`, or `None` for all annotated objects. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `Table` | One row per occurrence. |

#### read\_target\_table

```
read_target_table(
    record_ids: Iterable[EntityId] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> Table
```

Return every inline task target as one typed Arrow row.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `record_ids` | `Iterable[EntityId] | None` | Keep only targets of tasks whose inputs include one of these Records. | `None` |
| `split` | `Split | str | None` | Keep only targets of tasks of this split, or `None` for all. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `Table` | Targets in task ID and position order. |

#### read\_task\_table

```
read_task_table(
    record_ids: Iterable[EntityId] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> Table
```

Return the tasks as an Arrow table without building Task objects.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `record_ids` | `Iterable[EntityId] | None` | Keep only tasks whose inputs include one of these Records, or `None` for all. | `None` |
| `split` | `Split | str | None` | Keep only tasks of this split, or `None` for all. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `Table` | One row per task with public IDs in place of object references. Targets are in |
| `Table` | meth:`read_target_table`. |

#### read\_tasks

```
read_tasks(
    *,
    inputs: Iterable[Record] | None = None,
    split: Split | str | None = None,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    load: LoadPlan = ALL_LAZY,
) -> tuple[Task, ...]
```

Read the tasks that match every filter given, in one pass; no filter reads every task.

Every call builds new tasks. A record a task uses that is not among `inputs` is read once
and shared by every task of the call.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `inputs` | `Iterable[Record] | None` | Keep only tasks that use one of these Records as input, or `None` for every task. Tasks reuse these objects, which the read leaves as they are. | `None` |
| `split` | `Split | str | None` | Keep only tasks of this split, or `None` for every task. | `None` |
| `required_modalities` | `Iterable[InputModality] | None` | Keep only tasks that declare every one of these input modalities. | `None` |
| `supported_modalities` | `Iterable[InputModality] | None` | Keep only tasks whose declared input modalities are all among these. | `None` |
| `load` | `LoadPlan` | Which kinds of stored values to load into the objects the read builds. The default loads none. | `ALL_LAZY` |

Returns:

| Type | Description |
| --- | --- |
| `tuple[Task, ...]` | Concrete tasks in stable ID order. |

#### record\_ids

```
record_ids() -> tuple[EntityId, ...]
```

Return the ID of every Record, without building the Records.

The IDs come in the release's record order, which `iter_records()` and `read()` also use.
Every reader of a release returns the same order. In a release with imported Records, the
order follows where the values are stored, so that a walk over the records reads the stored
values in sequence. In any other release, the order is by ID.

Returns:

| Type | Description |
| --- | --- |
| `tuple[EntityId, ...]` | The IDs in the order :meth:`iter_records` yields them when called without arguments. |

#### verify

```
verify() -> None
```

Verify the size and checksum of every file the release reads.

These are the artifacts its manifest lists, and the files it pins on each payload source.

Raises:

| Type | Description |
| --- | --- |
| `TimeNetFileError` | If a file is missing, or its size or checksum differs from the manifest record that describes it: its own record, or the pin of a payload source file. |