Skip to content

Architecture

How TimeNet's packages, registries, and curation fit together. This page is the map; follow the links for per-component detail.

The big picture

TimeNet splits into three parts. A connector curates a raw source into a manifest plus parquet artifacts; the client/SDK reads the manifest from a registry and loads the data. The client never runs connector code.

What it is Ships Used by
timenet Python package TimeF format, reader/writer, registry client, engine, BaseConnector, SDK, CLI everyone (install coming soon)
registry a served location compiled manifests + parquet the SDK reads it; curation publishes to it
timenet-connectors a repo connector recipes + cards + the timenet-curate CLI connector authors (clone it)

There can be several registries: one public, private internal ones, or a local directory.

The two flows

PRODUCE  dataset.yaml + connector ─► engine (download -> convert -> derive_schema -> store) ─► publish ─┐
                                                                                                     ▼
                                                                                                 registry
CONSUME  SDK ─► get_manifest ─► fetch parquet ─► TimeFReader ─► Arrow  ◄──────────────────────────────┘

The compiled manifest.json (the card's human-authored metadata plus the schema derived from the data) is the single source of truth the SDK reads. Because the SDK never imports connector code, everything a consumer needs to interpret the parquet lives in the manifest.

Curation roles: connector, engine, curator

Three producer-side pieces, each with one job:

Role What it is Job
Connector one BaseConnector subclass per dataset (connectors) the dataset-specific recipe: download() fetches raw files, convert() builds a TimeFDataset. Knows nothing about the engine or registry.
Engine run_pipeline (curate & publish) drives any connector through the fixed pipeline and owns caching, idempotency, and force / clean_cache. Knows no dataset specifics.
Curator the timenet-curate CLI (curation) the entry point: resolves the id to its connector and runs the engine into a registry. 
timenet-curate build org/name                          curator
  └─ discovery.resolve("org/name") -> Connector class   (datasets/<org>/<name>/ exposes CONNECTOR)
      └─ run_pipeline(connector, <registry>)            engine
           metadata -> download -> convert -> derive_schema -> store -> <registry>/org/name/<version>/

metadata() reads the dataset.yaml card and store() streams through TimeFWriter. The output directory is itself a valid local registry, so the consume flow reads it straight back.

Where each component lives

Component Package Side
TimeF format, types, TimeFDataset, manifest timenet shared
CLI, SDK, registry client, TimeFReader timenet consumer
Engine, TimeFWriter, BaseConnector timenet producer
Connector recipes + cards, timenet-curate timenet-connectors producer

Design principles

  • The manifest is self-describing. The SDK reads schema, counts, and file pointers from manifest.json. It never runs connector code or globs the directory.
  • Types are plain frozen dataclasses. Specs, data sources, and annotations are frozen descriptors, so they pickle and round-trip through the reader with no runtime class synthesis. That keeps multiprocessing DataLoader workers safe.
  • Values are Arrow in, Arrow out. A TimeSeries exposes to_arrow() and to_numpy() over a private lazy loader. The writer stores float32 values as Parquet with BYTE_STREAM_SPLIT and zstd.
  • Units go through pint. One shared registry owns every definition and conversion.
  • Commits are atomic. The writer stages a version into a temp directory and publishes it with a single atomic rename. Once manifest.json is present, the version is committed.
  • Versions are immutable; edits are copy-on-write. Removing a row writes a new version through the same atomic path (edit_version); stable never-reused ids keep references valid, and content-defined chunking keeps the rewrite cheap on a deduplicating backend.

Lifecycle of a dataset

  1. Author a connector at datasets/<org>/<name>/ (its __init__.py exposes CONNECTOR) with its dataset.yaml card beside it, in timenet-connectors.
  2. Curate: timenet-curate build <org>/<name> runs the engine, compiles the manifest, and writes parquet.
  3. Verify locally by pointing the SDK at the output directory (itself a valid local registry).
  4. Publish the manifest + parquet to a registry.
  5. Consume: timenet download <id> reads the manifest and fetches the parquet.