Skip to content

Build & publish

Build turns a connector's raw source into a stored TimeF version. A version is a manifest.json, Parquet control tables, and a Parquet or Zarr values plane. TimeNet writes the version into a registry. Build runs on your machine. Today it publishes to a local registry. A hosted backend is planned. The command timenet-build build drives it. This page explains what happens underneath.

The pipeline

Each build runs in its own environment. Before the pipeline starts, timenet-build resolves the connector's requirements.txt (see Connectors). Then it re-runs itself under uv. The build environment layers those requirements over the same timenet and timenet-connectors that the parent runs. Two connectors that need incompatible libraries no longer collide. The manifest records the build environment as build_env.

To run the pipeline in the current interpreter instead, pass --no-isolation or set TIMENET_ISOLATION=off. Use this while you write a connector. The programmatic entry point timenet_connectors.build() works the same way. It is isolated by default. It runs in-process when TIMENET_ISOLATION=off.

The engine runs one connector through five stages in timenet.engine.run_pipeline:

from timenet.engine import run_pipeline

run_pipeline(
    connector, root, *,
    cache_dir=None, keep_cache=False, progress_cb=None, force=False,
)
  1. cache: create cache_dir. The default is <TIMENET_CACHE>/<dataset_id>.
  2. download: connector.download(cache_dir) fetches the raw references. Only this stage touches the network.
  3. convert: connector.convert(raw_refs) builds an in-memory TimeFDataset.
  4. derive_schema and store: the engine derives the schema first. It then calls store_dataset(), which streams the dataset through TimeFWriter and returns the committed version directory.
  5. clean: the engine removes cache_dir again. Pass keep_cache=True, or --keep-cache on the CLI, to keep the raw sources. A cache_dir you passed in yourself is never removed.

run_pipeline is idempotent. If a version is already committed, it short-circuits, unless you pass force=True. Distributed (Ray-backed) scheduling is out of scope for now.

Publishing

For a local registry, store is the publish step. The output directory is itself a valid local registry. WritableRegistry.store is the general primitive. The S3 and hosted backends will implement it. Publishing to those backends arrives when they do.

The authoring loop

Building a dataset follows one path:

  1. Add a connector at datasets/<org>/<name>/ in timenet-connectors. Its __init__.py exposes a BaseConnector as CONNECTOR.
  2. Put its dataset card, dataset.yaml, beside it. When the connector loads the card, TimeNet validates it against the packaged dataset-card.schema.json.
  3. Build it with timenet-build build.
  4. Verify the dataset: point the SDK at the output directory. The output directory is itself a valid local registry.
  5. When a hosted backend is available, publish the dataset.

See Connectors to learn how to write the download and convert steps. See the timenet.engine API for the full symbol listing.