Engine

run_pipeline(
    connector: BaseConnector,
    root: Path,
    *,
    cache_dir: Path | None = None,
    clean_cache: bool = False,
    progress_cb: Callable[[WriteProgressEvent], None]
    | None = None,
    force: bool = False,
) -> Path

Run one connector through the full curation pipeline and return the version directory.

Idempotent: if the target version is already committed, the expensive download / convert / store stages are skipped and the existing directory is returned. Pass force to rebuild it. Otherwise the stages are: create the cache directory, download raw references into it, convert them into a dataset, derive_schema, then store. The engine only writes local files; publishing to a remote registry is a separate step.

Parameters:

Name	Type	Description	Default
connector	BaseConnector	The connector to curate.	required
root	Path	Output root; the dataset is written to <root>/<dataset_id>/<version>/.	required
cache_dir	Path | None	Directory for downloaded artifacts (defaults to <TIMENET_CACHE>/<dataset_id>).	None
clean_cache	bool	Remove the cache directory once the dataset is stored. The raw sources are only needed during conversion, so this reclaims disk after a successful build (they re-download on the next run).	False
progress_cb	Callable[[WriteProgressEvent], None] | None	Optional writer progress callback.	None
force	bool	Rebuild even if the version is already committed.	False

Returns:

Type	Description
Path	The committed version directory.