Dataset

Build the axis describing a stream of time offsets.

Parameters:

Returns:

Attach several annotations to the record, all together or not at all.

The whole batch is validated before any of it is attached: if one annotation fails a check, the call raises and leaves the record unchanged. To keep the annotations before a failure attached, loop :meth:annotate instead.

Parameters:

Returns:

Raises:

Attach an annotation to the record and return it.

Parameters:

Returns:

Raises:

Build a :class:~timenet.types.TimeInterval between two wall-clock moments on this timeline.

The Record must have a known absolute timestamp.

Parameters:

Returns:

Build a :class:~timenet.types.TimePoint at a wall-clock moment on this record's timeline.

The Record must have a known absolute timestamp.

Parameters:

Returns:

Read the sole signal's values as an Arrow array, for the common single-signal record.

Returns:

Raises:

Read the sole signal's values as NumPy, without building Arrow when they are NumPy.

Returns:

Raises:

Yield annotation occurrences from this Record, its Sources, and its Signals.

Yield every signal in deterministic source and signal order.

Yield all sources in deterministic depth-first order.

Yields:

Raises:

Return the axis of a window that starts index values into this one.

This is exact at every rate, because the origin it moves is an index, not a derived time.

Parameters:

Returns:

Build the axis of a regularly sampled series from its exact rate.

This method does not accept a float. Every real sampling rate is a whole number of values per second, so write 500.0 as 500. A rate that is not whole has no single reading: 29.97 fps is 2997/100 by its spelling and 30000/1001 by its intent. Those two drift 3.6 ms apart over an hour. State which one with a :class:~fractions.Fraction, and build the Fraction from a string, not a float. Fraction(29.97) is the binary expansion (1054475631502295/35184372088832), and Fraction("29.97") is 2997/100.

Parameters:

Returns:

Raises:

Return the first value at or after a time offset.

Parameters:

Returns:

Return the time offset of one value, floored to whole microseconds.

This floor pairs with the ceiling in :meth:index_at_or_after. The two are exactly inverse at every period of one microsecond or coarser, including the non-integral ones.

Parameters:

Returns:

Attach one annotation and return it.

Parameters:

Returns:

Build an irregular series from materialized values and their time offsets.

The axis endpoints come from the stream itself, so the two cannot disagree. You cannot state a first or last time offset that the time offsets do not have. To convert wall-clock moments, use :func:~timenet.dataset.axis.time_offsets_from_datetimes before you call. Numeric NumPy values and normalized time offsets retain their buffers for NumPy reads, which return read-only arrays. Arrow conversion happens only on an Arrow read. Treat retained input buffers as read-only.

Parameters:

Returns:

Raises:

Build a signal whose values are loaded only when they are read.

loader and time_offsets_loader each take a value handle, which the signal keeps as it is, or a plain callable returning an Arrow array, which the signal calls on every read. A callable with its own read_steps() is rejected: subclass ArrayHandle to keep its range reads.

Returns:

Build a series from already-materialized values and hold them in memory in the spec's dtype.

Numeric NumPy arrays retain their buffers when dtype and layout already match the spec. NumPy reads return fresh read-only views; treat the input buffers as read-only while in use. Arrow conversion happens only on an Arrow read. Other inputs become Arrow arrays immediately. The array length sets n_values.

For files or remote sources, use :meth:from_loader and supply the length. That constructor does not read the values immediately.

Parameters:

Returns:

Return whether span applies to this series.

A step span selects only the series it counts on. A time span with no time_series selects every series, and a time span with time_series set selects only the series it lists. Series are compared by identity, not by name, so a span does not select a slice of a series it names: the slice is a new series.

This checks the scope only, not the time.

Parameters:

Returns:

Raises:

Return whether at least one value of this series falls inside span.

A value is inside when its time offset falls in [start_us, exclusive_end), the rule :meth:slice uses, so for a time interval slice(span, clip=True) keeps a value whenever this returns True. This compares times only and ignores the span's time_series: ask :meth:is_selected_by whether the span applies to this series.

Parameters:

Returns:

Raises:

Read a half-open temporal step range as Arrow without forcing a NumPy conversion.

A series whose stored values were loaded, or that holds its values in memory, is sliced. An unloaded stored series reads only the chunks that intersect the range, and one from a loader callable is read in full first.

Parameters:

Returns:

Raises:

Return a new lazy series that holds only the steps span covers.

The span must hold at least one value. A step span must count on this series. A time span is placed by time only, so ask :meth:is_selected_by first when the scope matters. The slice keeps the name, spec, metadata, and place on the timeline, drops annotations, and reads no values until it is read::

if signal.overlaps_with(interval):
    part = signal.slice(interval, clip=True)

The "Cut a window" docs show how to cut every signal of a record.

Parameters:

Returns:

Raises:

Return the half-open step range (start, stop) of this series that span covers.

The bridge to step-based forecasting libraries: stop - start is the horizon h that GluonTS, Nixtla, and fev speak in. The pair feeds :meth:read_steps to read the ground truth. A step span already counts in this series' own steps, so it is the range, bounded by the series' length. The axis locates a time span instead: the steps whose time offsets fall in [start_us, end_us), each rounded up to the next step. An ordinal series has no timeline, so a time span has no answer on it.

Parameters:

Returns:

Raises:

Read this series' per-value time offsets.

Only an irregular series has this. A regular axis computes its time offsets without a read, and an ordinal one has none. So neither has a stream to return.

Returns:

Raises:

Read the series' values as an Arrow array.

Returns:

Read the series' values as a read-only NumPy array.

Returns:

Raises:

Attach one annotation content item to the selected signals.

Timed and static annotations create one occurrence per selected Signal. Each occurrence shares the original content. All selected Signals are validated before any annotation is attached.

Returns:

Resolve an object or name selection against a source subtree.

Returns:

Raises:

Attach one annotation and return it.

Returns:

Select descendant signals by object or by human-readable name.

Returns:

Yield every descendant signal in deterministic source and signal order.

Yields:

Yield this source and all descendants in deterministic depth-first order.

Yields:

Raises:

Register and return a complete record.

The record and each of its signals get the next id of their kind.

Parameters:

Returns:

Validate and register one fully constructed task.

The Task owns its input and target object references. Registration verifies those objects against this dataset before it changes the dataset. It does not change the input Records. The task gets the next task id.

Parameters:

Returns:

Register several complete tasks, all together or not at all.

The method validates the whole batch before it attaches any task. If one task fails a check, the call raises an error, and the dataset and every task in the batch stay unchanged. To keep the tasks that passed before a failure, call :meth:add_task in a loop instead.

A task can derive from another task in the same batch. To do this, list the parent task in the deriving task's from_tasks field. The batch is checked as a unit, so the order of tasks within tasks does not matter.

Parameters:

Returns:

Attach a static annotation to the dataset.

Returns:

Raises:

Walk the dataset's instances and build its :class:DatasetSchema.

This method collects the distinct spec, annotation, and task types. It stores the result on the dataset, and it returns the result.

Returns:

Raises:

Print a plain-text summary of the dataset: its identity, counts, specs and columns, and a record preview.

This method works like pandas' describe and info methods. The preview reads only span metadata, not series values. The method checks value dtypes from one series per spec. This method works even before :meth:derive_schema runs, because it computes everything from the records.

Parameters:

Build a dataset from parts that are already constructed.

The reader uses this method when it reads a dataset back from disk.

Parameters:

Returns:

Return every task whatever its split, in insertion order.

Returns:

Return the tasks of one split, in insertion order.

A dataset partitions its tasks by setting :attr:~timenet.types.Task.split on each one. Records carry no split of their own: a record may serve tasks of several splits. A streamed dataset drains its stream and materializes the selection.

Parameters:

Returns:

Raises:

Return the union of selected splits without changing task assignments.

Parameters:

Returns:

Raises:

Return the tasks of the test split. See :meth:get_split.

Return the tasks of the training split. See :meth:get_split.

Return the tasks of the validation split. See :meth:get_split.

Copy Records, Signals, and tasks from a parent release into this dataset.

The parent must be a release that metadata.parents declares. The call records the parent in :attr:parent_cards and :attr:lineage. It does not change the card of this dataset.

• records= copies each Record whole, with its Sources, Signals, and annotations.
• signals= copies each Signal into a copy of its Record and of the Sources above it. Such a partial copy keeps the annotations that its Signals place.
• tasks= copies each task with the Records and Signals that it references and the tasks that it derives from. Each annotation that a task references points at the matching occurrence on the copy of its owner. If this dataset changed or removed that occurrence, the import attaches a copy of the parent's next to it.

Each copy is this dataset's own object, with new IDs and its own metadata and annotations, so edits do not change the parent. Its Signals keep the parent's value handles, so they read the parent's stored values unless load loads them. A write of this dataset reuses the parent's payloads, so it needs the parent release. Annotations on the parent dataset itself come only with a task that references them.

An import of an object whose copy this dataset holds reuses that copy, with this dataset's edits. If the call needs Signals that the copy of a Record lacks, it adds them to that copy, with the Sources and annotations that they bring. An object whose copy this dataset removed is copied again, under a new ID. An edit such as :func:~timenet.dataset.edit.remove_signals keeps the import state in the dataset it builds, so an import into that dataset works the same way. Imports are safe to repeat: if a call fails partway, repeating it completes it. Select many objects in one call, because each call reads them in batches. docs/timef-dataset.md describes the rules.

Parameters:

Returns:

Raises:

Yield the streamed tasks, validating each against the dataset before it is written.

The id of a streamed task is the task id counter's value at :meth:set_task_stream plus its position in the stream. Thus every pass over the stream gives the task the same id.

Streamed tasks skip :meth:add_task's checks, so validate each here as it passes through: an undeclared type, an attachment to an unknown record, a dangling reference, a bad answer, or an out-of-window span raises before the task reaches disk. The dataset holds no task list. Thus the cross-task checks (from_tasks derivations) that need every task at once do not run for a stream.

Yields:

Raises:

Yield the dataset's tasks, from the stream when one is set, else the materialized list.

Parameters:

Yields:

Raises:

Provide tasks as a re-iterable stream instead of materializing them in the dataset.

For a dataset with far more tasks than records (many questions over few recordings), holding every task in memory does not scale. A streaming connector builds the bounded records. Then it gives the tasks through source. The writer streams them to disk without a list.

Parameters:

Raises:

Return the tasks that take a record as input, optionally filtered by type.

Tasks reference their records, so this method scans the tasks for the ones whose inputs hold this record.

Parameters:

Returns:

Raises:

Return every task of a given type, in insertion order.

Parameters:

Returns:

Build an (X, y) training pair. By default, this method defers materialization.

This method requires every record to carry exactly one task of task. It pairs the values of that task's sole signal with the task's only scalar target. With split, only the tasks of that split count, and a record that has none of them is left out rather than rejected, so one call builds (X_train, y_train) and another (X_test, y_test). The features argument chooses the shape of X:

• "timestep" (the default): one feature per point, in a rectangular matrix. This needs equal-length records. The result is an Arrow FixedSizeListArray[T], or a NumPy (n, T) array of float32 values.
• "series": one sequence feature per record, so variable-length series work too. The result is an Arrow ListArray, or a NumPy (n,) object array of 1-D arrays.

With output="arrow" (the default), the method builds these arrays straight from the series loaders, with no NumPy copy in between. With output="numpy", the method materializes them. y is always the targets, as an Arrow string array or a 1-D NumPy array.

Parameters:

Returns:

Raises:

Write this dataset as one immutable TimeF version.

Parameters:

Returns: