Skip to content

Reader

timenet.reader source

Deserialization of a TimeF version directory into a :class:~timenet.dataset.TimeFDataset.

RECORD_BATCH_SIZE module-attribute source

RECORD_BATCH_SIZE = 1000

The most Records one batch of a Record stream holds, unless the stream sets another batch size.

TASK_BATCH_SIZE module-attribute source

TASK_BATCH_SIZE = 5000

The most tasks one batch of a task read or stream holds, unless a stream sets another batch size.

LoadPlan dataclass source

Specify, for each kind of stored value, whether a read loads it or leaves it lazy.

A read applies its plan to the objects it builds: a dataset's records, tasks, and annotations; a record's signals, time offsets, and annotations; a task's text, annotations, and the records, signals, and annotations it references. It loads nothing on objects passed to it, such as the records given to read_tasks, and nothing on tasks it builds only as another task's from_tasks. A plan never widens what a read reaches.

ALL_EAGER class-attribute source

ALL_EAGER: LoadPlan

Load every kind, so the objects a read builds need no storage afterwards.

ALL_LAZY class-attribute source

ALL_LAZY: LoadPlan

Load nothing. Every read's default.

signal_values class-attribute instance-attribute source

signal_values: Loading = Loading.LAZY

Signal arrays.

text class-attribute instance-attribute source

text: Loading = Loading.LAZY

Packed text: prompts, rationales, text targets, and annotation strings and string lists.

time_offsets class-attribute instance-attribute source

time_offsets: Loading = Loading.LAZY

Time offsets of irregular axes, one array per axis that its signals share.

Loading source

Bases: StrEnum

How a read delivers one kind of stored value on the objects it builds.

EAGER class-attribute instance-attribute source

EAGER = 'eager'

Loaded during the read and kept in memory in its value handle, so later reads don't touch storage.

LAZY class-attribute instance-attribute source

LAZY = 'lazy'

Read from storage when accessed.

TimeFReader source

Read a committed TimeF dataset through its DuckDB control plane.

metadata property source

metadata: DatasetMetadata

The dataset's descriptive metadata.

schema property source

schema: DatasetSchema

The dataset's type declaration.

values_backend property source

values_backend: str

The values-plane backend named by the manifest.

close source

close() -> None

Close cached handles while keeping the reader reusable.

get source

get(
    kind: type[E],
    id: EntityId,
    *,
    parent: ParentAlias | DatasetRef | None = None,
    load: LoadPlan = ALL_LAZY,
) -> E

Read one object by its ID, from this dataset or imported from a parent.

An ID is an opaque handle. It stays the same across edits of one dataset, and it means something only together with that dataset's reference: an object imported from a parent is found by the parent's ID, with parent naming the parent. An ID is not a lookup key for content; to find an object by what it is, read objects with :meth:iter_records or :meth:read_tasks and check their metadata.

get(Task, 7) reads the task with ID 7; get(AnswerTask, 7) also checks that it is an answer task. With parent, id is the ID the parent dataset gives the object, and the read returns this dataset's imported copy.

Parameters:

Name Type Description Default
kind type[E]

Record, Signal, or a task class such as Task or AnswerTask. Sources and annotations are read through the objects that hold them.

 required
id EntityId

The object's ID, a positive int64.

 required
parent ParentAlias | DatasetRef | None

The parent the object was imported from: the ParentAlias it was imported under, or its exact release as a DatasetRef.

 None
load LoadPlan

Which kinds of stored values to load into the object and what it references. The default loads none.

 ALL_LAZY

Returns:

Type Description
E

The object. A Signal is part of its Record, which the read builds too.

Raises:

Type Description
TimeFNotFoundError

If the ID or parent names nothing, or the stored object is not a kind.
TimeFValidationError

If kind is Source, Annotation, or another kind not listed above, id is not a positive int64, such as a string, or parent is neither a ParentAlias nor a DatasetRef, such as a plain string.
TimeFFormatError

If the catalog names no parent release or another one for the alias than the manifest does, or lists the alias or a parent ID more than once.

get_many source

get_many(
    kind: type[E],
    ids: Iterable[EntityId],
    *,
    parent: ParentAlias | DatasetRef | None = None,
    load: LoadPlan = ALL_LAZY,
) -> tuple[E, ...]

Read objects of one kind by their IDs, in one batch.

IDs are opaque handles, as :meth:get describes: stable across edits of this dataset, meaningful only with its reference, and no key for content. The objects form one graph: a Record that two of them reference is one object, and the Signals on one irregular axis share one time-offsets handle. load loads their values in one batch.

Parameters:

Name Type Description Default
kind type[E]

Record, Signal, or a task class such as Task or AnswerTask. Sources and annotations are read through the objects that hold them.

 required
ids Iterable[EntityId]

The objects' IDs, positive int64s in a list, tuple, or other iterable. With parent, the IDs the parent dataset gives the objects.

 required
parent ParentAlias | DatasetRef | None

The parent the objects were imported from: the ParentAlias they were imported under, or its exact release as a DatasetRef.

 None
load LoadPlan

Which kinds of stored values to load into the objects and what they reference. The default loads none.

 ALL_LAZY

Returns:

Type Description
tuple[E, ...]

One object per ID, in the order of ids. A repeated ID returns the same object.

Raises:

Type Description
TimeFNotFoundError

If an ID or parent names nothing, or a stored object is not a kind. Nothing is returned then.
TimeFValidationError

If kind is Source, Annotation, or another kind not listed above, ids is a string or holds anything but positive int64s, or parent is neither a ParentAlias nor a DatasetRef, such as a plain string.
TimeFFormatError

If the catalog names no parent release or another one for the alias than the manifest does, or lists the alias or a parent ID more than once.

iter_records source

iter_records(
    record_ids: Iterable[EntityId] | None = None,
    *,
    with_annotations: bool = True,
    load: LoadPlan = ALL_LAZY,
    batch_size: int = RECORD_BATCH_SIZE,
) -> Iterator[Record]

Yield complete records a bounded batch at a time, in a requested ID order or in record order.

The stream builds and loads each batch on its own, before it yields the batch. The records of a batch share the objects that the batch builds. So the signals on one irregular axis share one handle to the time offsets of that axis.

Batches share nothing. So after the stream yields the objects of a batch, it keeps none of them. It holds one batch of objects at a time. If two batches request one record ID, each batch builds that record as a separate object. The stream checks every requested ID before it builds the first batch.

Parameters:

Name Type Description Default
record_ids Iterable[EntityId] | None

IDs to read in result order, or None for every record in the order of :meth:record_ids.

 None
with_annotations bool

Whether to build annotations on the records and descendants.

 True
load LoadPlan

The kinds of stored values to load into the records that each batch builds. The default loads none.

 ALL_LAZY
batch_size int

The most records that a batch holds. By default, a batch holds up to 1,000 records (RECORD_BATCH_SIZE). The stream holds one batch at a time, with every value that the plan loaded. So with an eager plan, a smaller batch holds fewer arrays in memory.

 RECORD_BATCH_SIZE

Yields:

Type Description
Record

Records with loaded values for each kind of value that the plan names. Other stored
Record

values stay behind their value handles.

iter_tasks source

iter_tasks(
    *,
    inputs: Iterable[Record] | None = None,
    split: Split | str | None = None,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    load: LoadPlan = ALL_LAZY,
    batch_size: int = TASK_BATCH_SIZE,
) -> Iterator[Task]

Yield the tasks :meth:read_tasks reads, a bounded batch at a time.

Each batch is built and loaded on its own. Its tasks share the objects it builds: a record that two of them use is one object, and the signals on one irregular axis share one time-offsets handle. Batches share nothing but inputs, so the stream keeps none of a batch's objects once it has yielded them: it holds one batch of objects and that batch's task IDs at a time, and selects the next batch's IDs after the last one, while the reader keeps compact chunk locators for the signals it has resolved until it closes. A record or parent task that tasks of several batches use is built once per batch, as separate objects with the same ID, each loaded as far as its own batch's plan reaches it.

Parameters:

Name Type Description Default
inputs Iterable[Record] | None

Keep only tasks that use one of these Records as input, or None for every task. Tasks reference these Record objects directly, and the read leaves them as they are.

 None
split Split | str | None

Keep only tasks of this split, or None for every task. The filter runs in DuckDB, so a task outside the split is never built.

 None
required_modalities Iterable[InputModality] | None

Keep only tasks that declare every one of these input modalities.

 None
supported_modalities Iterable[InputModality] | None

Keep only tasks whose declared input modalities are all among these.

 None
load LoadPlan

Which kinds of stored values to load into the objects each batch builds. The default loads none. A batch builds but does not load what it reaches only through its tasks' from_tasks parents, which is the parents and their records and annotations; a record or annotation that a task of the batch uses as well loads.

 ALL_LAZY
batch_size int

The most tasks a batch holds. By default, a batch holds up to 5,000 tasks (TASK_BATCH_SIZE). Pass another value for larger or smaller batches. The stream holds one batch at a time, with every value its plan loaded, so with an eager plan a smaller batch holds fewer arrays in memory.

 TASK_BATCH_SIZE

Yields:

Type Description
Task

Concrete tasks in stable ID order.

open classmethod source

open(
    path: str | Path,
    *,
    sources_root: str | Path | None = None,
) -> TimeFReader

Open a local version directory for reading.

Opening reads only metadata, as :func:~timenet.registry.dependencies.open_local_version does. The reader opens the catalog on its first read. Close the reader after use, or use it as a context manager.

Parameters:

Name Type Description Default
path str | Path

The version directory, which holds manifest.json and control.duckdb.

 required
sources_root str | Path | None

The registry root to find payload sources under. By default this is the root that holds path as <root>/<org>/<name>/<version>.

 None

Returns:

Type Description
TimeFReader

A reader for the version.

Raises:

Type Description
FileNotFoundError

If path has no manifest.
TimeFValidationError

If the version has payload sources, is not stored as <root>/<org>/<name>/<version>, and no sources_root is given.
TimeNetMissingParentError

If a payload source is not under the sources root.

read source

read(*, load: LoadPlan = ALL_LAZY) -> TimeFDataset

Read the whole dataset, loading the kinds of stored values that load names.

Every call builds new objects. With the default, every stored value stays behind its value handle and reads through to storage on access. With LoadPlan.ALL_EAGER, the dataset needs no storage after the reader closes.

The dataset gets the parent cards and the lineage of the manifest, and the import map of the catalog, so a later import into it reuses the copies it holds instead of copying them again.

Parameters:

Name Type Description Default
load LoadPlan

Which kinds of stored values to load into the objects the read builds.

 ALL_LAZY

Returns:

Type Description
TimeFDataset

The dataset.

Raises:

Type Description
TimeFFormatError

If the catalog's parent aliases or their parent releases differ from the manifest's parent cards, or the catalog lists an alias or a parent ID more than once.

read_annotation_table source

read_annotation_table(
    object_type: ObjectKind | None = None,
) -> Table

Return every annotation occurrence with its content as an Arrow table.

Parameters:

Name Type Description Default
object_type ObjectKind | None

Keep occurrences on this object kind, such as ObjectKind.RECORD, or None for all annotated objects.

 None

Returns:

Type Description
Table

One row per occurrence.

read_target_table source

read_target_table(
    record_ids: Iterable[EntityId] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> Table

Return every inline task target as one typed Arrow row.

Parameters:

Name Type Description Default
record_ids Iterable[EntityId] | None

Keep only targets of tasks whose inputs include one of these Records.

 None
split Split | str | None

Keep only targets of tasks of this split, or None for all.

 None

Returns:

Type Description
Table

Targets in task ID and position order.

read_task_table source

read_task_table(
    record_ids: Iterable[EntityId] | None = None,
    *,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    split: Split | str | None = None,
) -> Table

Return the tasks as an Arrow table without building Task objects.

Parameters:

Name Type Description Default
record_ids Iterable[EntityId] | None

Keep only tasks whose inputs include one of these Records, or None for all.

 None
split Split | str | None

Keep only tasks of this split, or None for all.

 None

Returns:

Type Description
Table

One row per task with public IDs in place of object references. Targets are in
Table

meth:read_target_table.

read_tasks source

read_tasks(
    *,
    inputs: Iterable[Record] | None = None,
    split: Split | str | None = None,
    required_modalities: Iterable[InputModality]
    | None = None,
    supported_modalities: Iterable[InputModality]
    | None = None,
    load: LoadPlan = ALL_LAZY,
) -> tuple[Task, ...]

Read the tasks that match every filter given, in one pass; no filter reads every task.

Every call builds new tasks. A record a task uses that is not among inputs is read once and shared by every task of the call.

Parameters:

Name Type Description Default
inputs Iterable[Record] | None

Keep only tasks that use one of these Records as input, or None for every task. Tasks reuse these objects, which the read leaves as they are.

 None
split Split | str | None

Keep only tasks of this split, or None for every task.

 None
required_modalities Iterable[InputModality] | None

Keep only tasks that declare every one of these input modalities.

 None
supported_modalities Iterable[InputModality] | None

Keep only tasks whose declared input modalities are all among these.

 None
load LoadPlan

Which kinds of stored values to load into the objects the read builds. The default loads none.

 ALL_LAZY

Returns:

Type Description
tuple[Task, ...]

Concrete tasks in stable ID order.

record_ids source

record_ids() -> tuple[EntityId, ...]

Return the ID of every Record, without building the Records.

The IDs come in the release's record order, which iter_records() and read() also use. Every reader of a release returns the same order. In a release with imported Records, the order follows where the values are stored, so that a walk over the records reads the stored values in sequence. In any other release, the order is by ID.

Returns:

Type Description
tuple[EntityId, ...]

The IDs in the order :meth:iter_records yields them when called without arguments.

verify source

verify() -> None

Verify the size and checksum of every file the release reads.

These are the artifacts its manifest lists, and the files it pins on each payload source.

Raises:

Type Description
TimeNetFileError

If a file is missing, or its size or checksum differs from the manifest record that describes it: its own record, or the pin of a payload source file.