# Timenet.engine

## timenet.engine

The build engine: drive a connector through `download -> compose -> derive_schema -> store`.

### open\_parents

```
open_parents(
    metadata: DatasetMetadata, registry: LocalRegistry
) -> BuildContext
```

Open readers for the dataset's declared parents in a local registry.

The registry must hold every parent complete, with its payload sources. The engine never fetches
or builds a parent here.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `metadata` | `DatasetMetadata` | The child dataset card with exact parent references. | *required* |
| `registry` | `LocalRegistry` | The local registry containing the parents. | *required* |

Returns:

| Type | Description |
| --- | --- |
| `BuildContext` | The opened readers, keyed by parent alias. Close the context after the write. |

Raises:

| Type | Description |
| --- | --- |
| `TimeNetDatasetNotFoundError` | If the registry does not hold a declared parent. |

### run\_pipeline

```
run_pipeline(
    connector: BaseConnector,
    root: Path,
    *,
    cache_dir: Path | None = None,
    keep_cache: bool = False,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    force: bool = False,
    pinned_by: Mapping[DatasetRef, PayloadSource]
    | None = None,
) -> Path
```

Run one connector through the full build pipeline and return the version directory.

This function is idempotent. If the target version is already committed, it skips the expensive
`download`, `convert`, and `store` stages and returns the existing directory. Pass `force`
to rebuild it. Otherwise the stages run in order: create the cache directory, open the declared
parents from `root`, `download` raw references into the cache, `compose` them into a
dataset, check that the dataset has the card that the engine read, record each declared parent
in it, `derive_schema`, `store`, then delete the cache directory. Parents must already be in
`root`; the engine never fetches or builds them. A card that declares parents needs a
connector that overrides `compose`; otherwise the build raises before it opens a parent or
downloads anything. The engine only writes local files. Publishing to a remote registry is a
separate step.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `connector` | `BaseConnector` | The connector to build. | *required* |
| `root` | `Path` | Output root. This function writes the dataset to `<root>/<dataset_id>/<version>/`. | *required* |
| `cache_dir` | `Path | None` | Directory for downloaded artifacts (defaults to `<TIMENET_CACHE>/downloads/<dataset_id>`). | `None` |
| `keep_cache` | `bool` | Keep the cache directory instead of removing it once the dataset is stored. Conversion is the only stage that needs the raw sources, so removing them frees disk after a successful build. The sources re-download on the next run. | `False` |
| `progress_cb` | `Callable[[WriteProgressEvent], None] | None` | Optional writer progress callback. | `None` |
| `force` | `bool` | Rebuild even if the version is already committed. | `False` |
| `pinned_by` | `Mapping[DatasetRef, PayloadSource] | None` | The releases that already read files of this release, with their pins. The write publishes only if it reproduces each pinned file, and otherwise raises. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `Path` | The committed version directory. |

Raises:

| Type | Description |
| --- | --- |
| `TimeFValidationError` | If the card declares parents and the connector does not override `compose`, or the dataset that `compose` returns has another card than the one that the engine read. The build then stores no release. Also before any download, if `root` and the raw downloads that the build removes overlap, or the cache is the home folder or a folder above it. |

### store\_dataset

```
store_dataset(
    dataset: TimeFDataset,
    root: Path,
    *,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    values_backend: ValuesBackend = PARQUET,
    replace: bool = False,
    pinned_by: Mapping[DatasetRef, PayloadSource]
    | None = None,
) -> Path
```

Serialize a populated dataset to the TimeF format under `root`.

If the dataset has no schema, this function derives it first. It then streams the dataset through a
:class:`~timenet.writer.TimeFWriter`. This function lives on the engine, not on
:class:`~timenet.connectors.BaseConnector`, because it reads only `dataset`. This keeps the
connector contract at fetch-and-convert and avoids a connector-to-writer dependency.

Parameters:

| Name | Type | Description | Default |
| --- | --- | --- | --- |
| `dataset` | `TimeFDataset` | The populated dataset from `convert`. | *required* |
| `root` | `Path` | Parent directory. This function creates the version directory beneath it. | *required* |
| `progress_cb` | `Callable[[WriteProgressEvent], None] | None` | Optional writer progress callback. | `None` |
| `values_backend` | `ValuesBackend` | Storage backend for the values plane. | `PARQUET` |
| `replace` | `bool` | Replace the version if it is already committed, keeping it until the new one is published. | `False` |
| `pinned_by` | `Mapping[DatasetRef, PayloadSource] | None` | The releases that already read files of this release, with their pins. The writer publishes only if it reproduces each pinned file. | `None` |

Returns:

| Type | Description |
| --- | --- |
| `Path` | The committed version directory. |