Engine

timenet.engine ¶source

The build engine: drive a connector through download -> convert -> derive_schema -> store.

publish_pipeline ¶source

publish_pipeline(
    connector: BaseConnector,
    registry: WritableRegistry,
    *,
    cache_dir: Path | None = None,
    keep_cache: bool = False,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    force: bool = False,
    values_backend: str | None = None,
) -> str

Run one connector and publish the result through a writable registry.

Like :func:run_pipeline, but the converted dataset is handed to registry.store instead of written to a local directory, so it also targets a remote or S3 registry. It is idempotent: an already-committed version skips the download and convert stages and returns unless force.

Parameters:

Name	Type	Description	Default
`connector`	`BaseConnector`	The connector to build.	required
`registry`	`WritableRegistry`	The writable registry to publish into (local, remote, or S3).	required
`cache_dir`	`Path \| None`	Directory for downloaded artifacts (defaults to `<TIMENET_CACHE>/<dataset_id>`).	`None`
`keep_cache`	`bool`	Keep the cache directory instead of removing it after publishing.	`False`
`values_backend`	`str \| None`	Storage backend for the values plane (`"parquet"` or `"zarr"`). When `None`, this function uses the connector's `values_backend`.	`None`
`progress_cb`	`Callable[[WriteProgressEvent], None] \| None`	Optional writer progress callback.	`None`
`force`	`bool`	Republish even if the version is already committed.	`False`

Returns:

Type	Description
`str`	The published version string.

run_pipeline ¶source

run_pipeline(
    connector: BaseConnector,
    root: Path,
    *,
    cache_dir: Path | None = None,
    keep_cache: bool = False,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    force: bool = False,
    values_backend: str | None = None,
) -> Path

Run one connector through the full build pipeline and return the version directory.

This function is idempotent. If the target version is already committed, it skips the expensive download, convert, and store stages and returns the existing directory. Pass force to rebuild it. Otherwise the stages run in order: create the cache directory, download raw references into it, convert them into a dataset, derive_schema, store, then delete the cache directory. The engine only writes local files. Publishing to a remote registry is a separate step.

Parameters:

Name	Type	Description	Default
`connector`	`BaseConnector`	The connector to build.	required
`root`	`Path`	Output root. This function writes the dataset to `<root>/<dataset_id>/<version>/`.	required
`cache_dir`	`Path \| None`	Directory for downloaded artifacts (defaults to `<TIMENET_CACHE>/<dataset_id>`).	`None`
`keep_cache`	`bool`	Keep the cache directory instead of removing it once the dataset is stored. Conversion is the only stage that needs the raw sources, so removing them frees disk after a successful build. The sources re-download on the next run.	`False`
`values_backend`	`str \| None`	Storage backend for the values plane (`"parquet"` or `"zarr"`). When `None`, this function uses the connector's `values_backend`, so a connector that needs Zarr declares it once on the class.	`None`
`progress_cb`	`Callable[[WriteProgressEvent], None] \| None`	Optional writer progress callback.	`None`
`force`	`bool`	Rebuild even if the version is already committed.	`False`

Returns:

Type	Description
`Path`	The committed version directory.

store_dataset ¶source

store_dataset(
    dataset: TimeFDataset,
    root: Path,
    *,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    values_backend: str = "parquet",
) -> Path

Serialize a populated dataset to the TimeF format under root.

If the dataset has no schema, this function derives it first. It then streams the dataset through a :class:~timenet.writer.TimeFWriter. This function lives on the engine, not on :class:~timenet.connectors.BaseConnector, because it reads only dataset. This keeps the connector contract at fetch-and-convert and avoids a connector-to-writer dependency.

Parameters:

Name	Type	Description	Default
`dataset`	`TimeFDataset`	The populated dataset from `convert`.	required
`root`	`Path`	Parent directory. This function creates the version directory beneath it.	required
`progress_cb`	`Callable[[WriteProgressEvent], None] \| None`	Optional writer progress callback.	`None`
`values_backend`	`str`	Storage backend for the values plane.	`'parquet'`

Returns:

Type	Description
`Path`	The committed version directory.