Skip to content

Connectors

timenet.connectors source

The connector contract: :class:BaseConnector.

BaseConnector source

Bases: ABC, Generic[TRaw]

Abstract base for dataset connectors. One concrete subclass per dataset.

A connector lives in its own folder and declares its descriptive identity in a dataset.yaml card beside it (read by :meth:metadata). Set :attr:CARD to point elsewhere. Subclasses implement a download stage and either convert or compose. Connectors take no constructor arguments.

CARD class-attribute source

CARD: str | Path | None = None

Optional explicit path to the dataset card YAML. When None (the default), the connector reads the card from dataset.yaml in its own folder.

values_backend class-attribute instance-attribute source

values_backend: str = 'parquet'

Default storage backend for this connector's values plane.

compose source

compose(
    raw_refs: list[TRaw], context: BuildContext
) -> TimeFDataset

Build the dataset from raw references and the opened parent releases.

A connector whose card declares parents must override this hook. The connector reads each parent through context.parents[alias] and copies objects with dataset.import_from(...). After this hook returns, the engine records each declared parent in the dataset: its card and its lineage. The default ignores context and calls :meth:convert.

Parameters:

Name Type Description Default
raw_refs list[TRaw]

The references returned by :meth:download.

 required
context BuildContext

The opened parents, keyed by the parent aliases on the card.

 required

Returns:

Type Description
TimeFDataset

The populated dataset.

convert source

convert(raw_refs: list[TRaw]) -> TimeFDataset

Parse raw references and populate a :class:~timenet.dataset.TimeFDataset.

CPU-bound: no network I/O. Attach time-series values as lazy loaders instead of materializing them.

Parameters:

Name Type Description Default
raw_refs list[TRaw]

The references returned by :meth:download.

 required

Returns:

Type Description
TimeFDataset

The populated dataset.

Raises:

Type Description
NotImplementedError

If called on a connector that only implements compose.

download source

download(cache_dir: Path) -> list[TRaw]

Fetch or discover raw source files and return lightweight references to them.

I/O only: no parsing, no array work. Must be idempotent for a given cache_dir. Override this for a synchronous connector. For an I/O-bound one, override :meth:download_async instead and leave this default, which drives it to completion, since the engine calls connectors synchronously.

Parameters:

Name Type Description Default
cache_dir Path

Directory to write downloaded files into (created by the engine).

 required

Returns:

Type Description
list[TRaw]

Raw references passed directly to :meth:convert.

download_async async source

download_async(cache_dir: Path) -> list[TRaw]

Async variant of :meth:download for connectors whose downloads are I/O-bound.

Override this to fetch artifacts concurrently, for example with the connector HTTP download helpers. The default :meth:download runs it for you. Implement exactly one of the two.

Parameters:

Name Type Description Default
cache_dir Path

Directory to write downloaded files into (created by the engine).

 required

Returns:

Type Description
list[TRaw]

Raw references passed directly to :meth:convert.

Raises:

Type Description
NotImplementedError

If a subclass overrides neither :meth:download nor :meth:download_async.

metadata source

metadata() -> DatasetMetadata

Return the dataset's descriptive identity, loaded and validated from its card YAML.

Reads the card by convention (dataset.yaml beside the connector, unless :attr:CARD overrides it). Its dataset_id must match the id used to register or build the connector.

Returns:

Type Description
DatasetMetadata

The dataset's :class:~timenet.types.DatasetMetadata.

BuildContext dataclass source

The declared parent releases of one connector build, opened by parent alias.

The engine opens the parents from the output registry before the download starts, and records each of them in the composed dataset after :meth:~timenet.connectors.BaseConnector.compose returns. The context only holds the readers and closes them.

parents class-attribute instance-attribute source

parents: dict[ParentAlias, TimeFReader] = field(
    default_factory=dict
)

references class-attribute instance-attribute source

references: dict[ParentAlias, DatasetRef] = field(
    default_factory=dict
)

close source

close() -> None

Release the parent readers once the writer has consumed them.

The reader opened last closes first. A reader that fails to close does not keep the others open: each of them still closes, and this method then raises the error.