Registry¶

A registry serves compiled TimeF versions to the SDK. A compiled TimeF version has a manifest, an immutable DuckDB control database, and a values plane in Parquet or Zarr format. A registry never runs connector code. The registry code lives in timenet.registry.

You can have several registries: a public registry, private internal registries, or a local directory. The output of build is itself a valid local registry.

Choosing a registry¶

To read a dataset, use TimeNet.open() with its ID, or TimeFReader.open() with a version directory.

from timenet.registry import open_registry

registry = open_registry("./local_registry")

open_registry(uri) dispatches by scheme:

An unrecognized scheme, for example gs:// or az://, raises ValueError instead of becoming a local path. open_registry expands ~ in a local path or a file:// URI.

Remote registries¶

http(s):// and timenet:// both select RemoteRegistry. timenet:// is a shorthand for the hosted https://registry.timenet.ai service root and takes no path: dataset ids are passed to the client methods, not folded into the registry URI. An http(s):// URI is used verbatim as the service root, which is how you point at a private or a dev deployment.

The HTTP transport authenticates with a bearer token read from $TIMENET_TOKEN. When it is unset the SDK talks to the registry anonymously (no Authorization header), which is enough for public reads. Set it to reach private datasets or to write:

export TIMENET_TOKEN=your-token

S3 registries¶

An s3://bucket/prefix URI opens an S3Registry. boto3 reads credentials, region, and an optional endpoint override from the environment (AWS_* vars, AWS_PROFILE, AWS_ENDPOINT_URL), so nothing is hardcoded. Install the extra:

uv add 'timenet[s3]'

The layout under the prefix mirrors the local one, org/name/version/… with manifest.json beside the data files. store uploads to a temporary version.tmp-<uuid> prefix and moves each object into place (manifest.json last), so a failed publish never leaves a partial version. open reads lazily and load reads every value. Both reuse complete local copies where they exist. Otherwise they read remote payload ranges through a pyarrow S3FileSystem. The backend has no catalog, so list and search are unsupported: publish and fetch by an explicit org/name@version.

BaseRegistry¶

Every backend implements this contract: four data-access methods and two shared methods, download_version and search.

LocalRegistry serves a <root>/<dataset_id>/<version>/ tree. RemoteRegistry serves the same layout over the versioned REST contract (GET /api/v1/datasets, /api/v1/datasets/{id}/{version}/manifest, .../download/{relpath}); S3Registry serves the same layout under an s3://bucket/prefix root. The S3 backend has no catalog, so list_datasets and search raise NotImplementedError.

A dataset id is an org/name pair, for example chengsenwang/tsqa. The id nests one level deep on disk, at <root>/chengsenwang/tsqa/<version>/. list_datasets finds these ids at any depth. Use lowercase ids to avoid casing clashes on case-insensitive filesystems.

Payload sources¶

A version can reuse payload files that other versions store: its payload sources. Its manifest names each one and pins the files it reads from it (see Shared payloads). A registry serves a version together with its payload sources:

• open_version opens each payload source from the same registry and checks it against the pins, reading only manifests and, for local files, file sizes. A remote registry prefers complete cached copies and otherwise reads remotely. See Opening a version for the checks and their errors.
• download_version downloads complete copies of the version and of each payload source, and checks each source against the pins. If a downloaded file has another size or checksum than its manifest records, the download raises TimeNetFileError and keeps no partial copy.
• store refuses a version whose payload sources the registry does not hold, holds in another form, or that read payloads of the version itself. The writer checks this at the first reuse of each source, and raises the errors that opening raises for a missing or differing source.

A version lists every payload source it reads from, so a registry resolves only those, never the sources' own sources.

A client of a local registry prepares the version and its payload sources before a read. For each of these releases, the client reuses a complete copy, or copies the release from fetch_from. If auto_build is set, the client can also build the release. If the client cannot get a release, the read raises an error. The client builds a payload source with its own connector, at the exact version of the source.

If fetch_from is set, the client repairs a damaged copy from that registry. Without fetch_from, the read raises TimeNetFileError for a damaged copy. A remote client never runs connector code. It rejects fetch_from. See Open, load, or download.

Remote reads of payload sources¶

The open_version() method of an HTTP or S3 registry resolves the manifests of the payload sources before it returns. Reads then use complete cached copies where they exist. Otherwise they read byte ranges of the remote files, as each read needs them. The first read of Records or tasks downloads the catalog of the version and makes sure that it matches the manifest. A read never opens the catalog of a parent release.

HTTP reads use the range reader and the read-ahead cache of fsspec. The object server must support byte ranges. The client requests a new signed URL after one expires. The registry token goes only with API requests.

A read of byte ranges cannot make sure that a whole file matches its checksum. download() and download_version() copy complete files, and make sure that each file has its recorded size and checksum. TimeFReader.verify() reads and hashes every file that a release reads.

Writing to a registry¶

A WritableRegistry adds one write method to the read contract. As a result, build can publish into any backend, not only a local directory:

LocalRegistry implements store by streaming the dataset through a TimeFWriter, which stages under <version>.tmp-* and publishes with a single atomic rename. With force, the old version stays in place until the new one is complete. A replace is not isolated from readers that already opened the old version, or from versions that reuse its payloads, which then fail to open. RemoteRegistry runs the publish flow below. S3Registry uploads to a temporary prefix and moves each object into place (manifest.json last). Each backend passes itself to the writer, which checks every payload source in that registry while it stages the release locally. open_writable_registry(uri) resolves a URI like open_registry but returns a WritableRegistry.

from timenet.registry import open_writable_registry

registry = open_writable_registry("./local_registry")
version = registry.store(dataset)   # schema derived if needed, atomic commit

Every backend is a WritableRegistry. As a result, this type alone does not show if a backend is a directory the engine can write to or a remote stub. local_registry_path(uri) gives this information. It returns the directory that a file:// URI or a plain path names. It raises TimeNetRegistryError for a remote scheme. default_registry_path() uses local_registry_path to find the default local registry. It uses $TIMENET_REGISTRY when its value is a local path. Otherwise, it uses <home>/registry. This is how timenet-build build resolves its output when --out is absent. The timenet_connectors.builder and load helpers use it too.

Remote publishing¶

RemoteRegistry.store compiles the dataset locally, then hands the artifacts to the service over the REST contract. Dataset bytes never pass through the API: the service issues a presigned PUT per file and the SDK uploads straight to the object store. The token must be a writer token ($TIMENET_TOKEN); an anonymous or read-only caller is rejected by the service.

1. Compile into a temporary staging directory with a TimeFWriter, deriving the schema first if the dataset has none. An already-committed version returns early unless force. The writer checks every payload source against the service, so a version whose sources the service does not hold is refused before anything is published.
2. POST /datasets/{id}/{version}/publish with the compiled manifest.json as the body. The service registers the pending version and returns the list of files it expects.
3. For each file, POST /datasets/{id}/{version}/publish/upload-url with {"path": relpath} to get a presigned grant ({"url", "headers"}), then PUT the file's bytes to that URL with the returned headers.
4. POST /datasets/{id}/{version}/finalize to commit the version. The staging directory is removed whether or not the publish succeeds.

from timenet.registry import open_writable_registry

registry = open_writable_registry("timenet://")  # needs $TIMENET_TOKEN
version = registry.store(dataset)                 # compile, upload, finalize

search¶

registry.search(
    query=None, domain=None, task=None, license=None,
    time_series_spec=None, dataset_id=None, tag=None, limit=100,
)

Every filter takes a scalar value or a list. The registry ignores None filters and combines all non-None filters with AND logic. The consumer CLI, timenet search, mirrors this behavior exactly. limit caps the number of results, with a default of 100. limit=0 returns no results. A negative limit raises ValueError.

The type-filters, task and time_series_spec, resolve each dataset's schema from its committed manifest. The manifest always carries the derived schema, so the registry does not need a precomputed_schema.

See the API reference for timenet.registry for the full symbol listing.