Timenet.engine

timenet.engine ¶source

The build engine: drive a connector through download -> compose -> derive_schema -> store.

open_parents ¶source

open_parents(
    metadata: DatasetMetadata, registry: LocalRegistry
) -> BuildContext

Open readers for the dataset's declared parents in a local registry.

The registry must hold every parent complete, with its payload sources. The engine never fetches or builds a parent here.

Parameters:

Name	Type	Description	Default
`metadata`	`DatasetMetadata`	The child dataset card with exact parent references.	required
`registry`	`LocalRegistry`	The local registry containing the parents.	required

Returns:

Type	Description
`BuildContext`	The opened readers, keyed by parent alias. Close the context after the write.

Raises:

Type	Description
`TimeNetDatasetNotFoundError`	If the registry does not hold a declared parent.

run_pipeline ¶source

run_pipeline(
    connector: BaseConnector,
    root: Path,
    *,
    cache_dir: Path | None = None,
    keep_cache: bool = False,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    force: bool = False,
    pinned_by: Mapping[DatasetRef, PayloadSource]
    | None = None,
) -> Path

Run one connector through the full build pipeline and return the version directory.

This function is idempotent. If the target version is already committed, it skips the expensive download, convert, and store stages and returns the existing directory. Pass force to rebuild it. Otherwise the stages run in order: create the cache directory, open the declared parents from root, download raw references into the cache, compose them into a dataset, check that the dataset has the card that the engine read, record each declared parent in it, derive_schema, store, then delete the cache directory. Parents must already be in root; the engine never fetches or builds them. A card that declares parents needs a connector that overrides compose; otherwise the build raises before it opens a parent or downloads anything. The engine only writes local files. Publishing to a remote registry is a separate step.

Parameters:

Name	Type	Description	Default
`connector`	`BaseConnector`	The connector to build.	required
`root`	`Path`	Output root. This function writes the dataset to `<root>/<dataset_id>/<version>/`.	required
`cache_dir`	`Path \| None`	Directory for downloaded artifacts (defaults to `<TIMENET_CACHE>/downloads/<dataset_id>`).	`None`
`keep_cache`	`bool`	Keep the cache directory instead of removing it once the dataset is stored. Conversion is the only stage that needs the raw sources, so removing them frees disk after a successful build. The sources re-download on the next run.	`False`
`progress_cb`	`Callable[[WriteProgressEvent], None] \| None`	Optional writer progress callback.	`None`
`force`	`bool`	Rebuild even if the version is already committed.	`False`
`pinned_by`	`Mapping[DatasetRef, PayloadSource] \| None`	The releases that already read files of this release, with their pins. The write publishes only if it reproduces each pinned file, and otherwise raises.	`None`

Returns:

Type	Description
`Path`	The committed version directory.

Raises:

Type	Description
`TimeFValidationError`	If the card declares parents and the connector does not override `compose`, or the dataset that `compose` returns has another card than the one that the engine read. The build then stores no release. Also before any download, if `root` and the raw downloads that the build removes overlap, or the cache is the home folder or a folder above it.

store_dataset ¶source

store_dataset(
    dataset: TimeFDataset,
    root: Path,
    *,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    values_backend: ValuesBackend = PARQUET,
    replace: bool = False,
    pinned_by: Mapping[DatasetRef, PayloadSource]
    | None = None,
) -> Path

Serialize a populated dataset to the TimeF format under root.

If the dataset has no schema, this function derives it first. It then streams the dataset through a :class:~timenet.writer.TimeFWriter. This function lives on the engine, not on :class:~timenet.connectors.BaseConnector, because it reads only dataset. This keeps the connector contract at fetch-and-convert and avoids a connector-to-writer dependency.

Parameters:

Name	Type	Description	Default
`dataset`	`TimeFDataset`	The populated dataset from `convert`.	required
`root`	`Path`	Parent directory. This function creates the version directory beneath it.	required
`progress_cb`	`Callable[[WriteProgressEvent], None] \| None`	Optional writer progress callback.	`None`
`values_backend`	`ValuesBackend`	Storage backend for the values plane.	`PARQUET`
`replace`	`bool`	Replace the version if it is already committed, keeping it until the new one is published.	`False`
`pinned_by`	`Mapping[DatasetRef, PayloadSource] \| None`	The releases that already read files of this release, with their pins. The writer publishes only if it reproduces each pinned file.	`None`

Returns:

Type	Description
`Path`	The committed version directory.