# Bases ## timenet\_connectors.bases Reusable connector base classes. ### huggingface A reusable base for connectors that pull rows from a HuggingFace Hub dataset. Subclasses set `HF_REPO`, ship a `dataset.yaml` card beside the connector, and implement `convert()`. Real downloads read the Hub's auto-generated parquet ref, which the Hub produces for public datasets (gated ones included). `huggingface_hub` is imported lazily so base users don't need it (install the `huggingface` extra) and reads `HF_TOKEN` from the environment, so gated datasets work with no extra wiring. Fully private datasets have no auto-parquet ref and aren't supported by this base. #### BaseHuggingFaceConnector Bases: `BaseConnector[dict[str, Any]]`, `ABC` Base class for HuggingFace-backed connectors. Rows are plain dicts (one per dataset row). ##### CARD `class-attribute` ``` CARD: str | Path | None = None ``` Optional explicit path to the dataset card YAML. When `None` (the default), the card is read from `dataset.yaml` in the connector's own folder. ##### HF\_REPO `class-attribute` ``` HF_REPO: str ``` ##### PARQUET\_REVISION `class-attribute` ``` PARQUET_REVISION: str = 'refs/convert/parquet' ``` ##### convert `abstractmethod` ``` convert(raw_refs: list[TRaw]) -> TimeFDataset ``` Parse raw references and populate a :class:`~timenet.dataset.TimeFDataset`. CPU-bound: no network I/O. Time-series values are attached as lazy loaders, not materialized. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `raw_refs` | `list[TRaw]` | The references returned by :meth:`download`. | *required* | Returns: | Type | Description | | --- | --- | | `TimeFDataset` | The populated dataset. | ##### download ``` download(cache_dir: Path) -> list[dict[str, Any]] ``` Download the repo's auto-converted parquet file(s) and return their rows. Reads the Hub's `refs/convert/parquet` branch (see :attr:`PARQUET_REVISION`) so any source format is handled uniformly. For very large datasets the Hub's conversion can be partial. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `cache_dir` | `Path` | Where Hub files are cached. | *required* | Returns: | Type | Description | | --- | --- | | `list[dict[str, Any]]` | One dict per row across all parquet files. | Raises: | Type | Description | | --- | --- | | `ImportError` | If the `huggingface` extra (`huggingface_hub`) is not installed. | ##### metadata ``` metadata() -> DatasetMetadata ``` Return the dataset's descriptive identity, loaded and validated from its card YAML. Reads the card by convention (`dataset.yaml` beside the connector, unless :attr:`CARD` overrides it). Its `dataset_id` must match the id the connector is registered/curated under. Returns: | Type | Description | | --- | --- | | `DatasetMetadata` | The dataset's :class:`~timenet.types.DatasetMetadata`. | ##### store ``` store( dataset: TimeFDataset, root: Path, *, progress_cb: Callable[[WriteProgressEvent], None] | None = None, ) -> Path ``` Serialize a populated dataset to the TimeF format under `root`. Derives the schema first if the dataset has none, then streams it through a :class:`~timenet.writer.TimeFWriter`. Most connectors do not override this. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `dataset` | `TimeFDataset` | The populated dataset from :meth:`convert`. | *required* | | `root` | `Path` | Parent directory; the version directory is created beneath it. | *required* | | `progress_cb` | `Callable[[WriteProgressEvent], None] | None` | Optional writer progress callback. | `None` | Returns: | Type | Description | | --- | --- | | `Path` | The committed version directory. | ### physionet A reusable base for connectors that read WFDB records from PhysioNet. Subclasses download a PhysioNet database archive (a zip served from PhysioNet's open `physionet-open` S3 bucket, or any HTTP URL) and read its records with `wfdb `\_. `wfdb` and `boto3` are imported lazily so base users who only curate offline datasets don't need them (install the `physionet` extra); a missing library raises an actionable error. #### TRaw `module-attribute` ``` TRaw = TypeVar('TRaw') ``` #### BasePhysioNetConnector Bases: `BaseConnector[TRaw]`, `ABC` Base class for PhysioNet-backed connectors: WFDB record I/O plus archive caching. ##### CARD `class-attribute` ``` CARD: str | Path | None = None ``` Optional explicit path to the dataset card YAML. When `None` (the default), the card is read from `dataset.yaml` in the connector's own folder. ##### convert `abstractmethod` ``` convert(raw_refs: list[TRaw]) -> TimeFDataset ``` Parse raw references and populate a :class:`~timenet.dataset.TimeFDataset`. CPU-bound: no network I/O. Time-series values are attached as lazy loaders, not materialized. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `raw_refs` | `list[TRaw]` | The references returned by :meth:`download`. | *required* | Returns: | Type | Description | | --- | --- | | `TimeFDataset` | The populated dataset. | ##### download ``` download(cache_dir: Path) -> list[TRaw] ``` Fetch or discover raw source files and return lightweight references to them. I/O only: no parsing, no array work. Must be idempotent for a given `cache_dir`. Defaults to returning no references, so a synthetic connector that builds everything in :meth:`convert` need not override it. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `cache_dir` | `Path` | Directory to write downloaded files into (created by the engine). | *required* | Returns: | Type | Description | | --- | --- | | `list[TRaw]` | Raw references passed directly to :meth:`convert` (empty by default). | ##### metadata ``` metadata() -> DatasetMetadata ``` Return the dataset's descriptive identity, loaded and validated from its card YAML. Reads the card by convention (`dataset.yaml` beside the connector, unless :attr:`CARD` overrides it). Its `dataset_id` must match the id the connector is registered/curated under. Returns: | Type | Description | | --- | --- | | `DatasetMetadata` | The dataset's :class:`~timenet.types.DatasetMetadata`. | ##### store ``` store( dataset: TimeFDataset, root: Path, *, progress_cb: Callable[[WriteProgressEvent], None] | None = None, ) -> Path ``` Serialize a populated dataset to the TimeF format under `root`. Derives the schema first if the dataset has none, then streams it through a :class:`~timenet.writer.TimeFWriter`. Most connectors do not override this. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `dataset` | `TimeFDataset` | The populated dataset from :meth:`convert`. | *required* | | `root` | `Path` | Parent directory; the version directory is created beneath it. | *required* | | `progress_cb` | `Callable[[WriteProgressEvent], None] | None` | Optional writer progress callback. | `None` | Returns: | Type | Description | | --- | --- | | `Path` | The committed version directory. | ### s3 A small, reusable S3 download helper for connectors. Downloads an `s3://bucket/key` object to a local path via boto3 (multipart/parallel). Credentials come from the environment / boto3's default chain when present, and fall back to anonymous (unsigned) requests otherwise, so public buckets such as PhysioNet's `physionet-open` work with no credentials. `boto3` is imported lazily so base users who only curate offline datasets don't need it. #### download\_s3\_object ``` download_s3_object(s3_url: str, dest: Path) -> None ``` Download an `s3://bucket/key` object to `dest`, creating parent directories. Parameters: | Name | Type | Description | Default | | --- | --- | --- | --- | | `s3_url` | `str` | The object URL, `s3:///`. | *required* | | `dest` | `Path` | The local destination path. | *required* | Raises: | Type | Description | | --- | --- | | `ValueError` | If `s3_url` is not an `s3://` URL carrying both a bucket and a key. |