Skip to content

Timenet connectors.sources

timenet_connectors.sources source

Shared source access and media conversion helpers for dataset connectors.

huggingface_hub source

Helpers for downloading files from pinned Hugging Face dataset revisions.

:func:hub_snapshot calls huggingface_hub.snapshot_download for the files that match the given patterns, such as a filename or data/*.parquet. The Hub client runs the downloads in its own thread pool. Its progress bars are hidden; the file bytes go to the :mod:~timenet_connectors.download.progress sink, so the build CLI shows them like any other download. huggingface_hub is listed in the connector's requirements.txt and imported lazily, so --no-isolation still works. The Hub client reads HF_TOKEN from the environment for gated datasets.

hub_snapshot async source

hub_snapshot(
    repo: str,
    revision: str,
    cache_dir: Path,
    patterns: Sequence[str],
) -> Path

Download the files of a dataset revision that match patterns.

Parameters:

Name Type Description Default
repo str

The Hub dataset id, such as "BASH-Lab/SensorCaps".

 required
revision str

The revision to read, normally a pinned commit.

 required
cache_dir Path

Directory where the connector caches Hub files.

 required
patterns Sequence[str]

Repo-relative filenames or globs, such as "data/*.parquet".

 required

Returns:

Type Description
Path

The local snapshot directory, with the files at their repo paths. The caller checks
Path

that the files it needs exist.

Raises:

Type Description
ImportError

If huggingface_hub, declared in the connector's requirements, is missing.
TimeNetDownloadError

If the download fails.

images source

Represent source images as RGB or RGBA tensor Signals for Zarr.

image_signal source

image_signal(source: Path | bytes, *, name: str) -> Signal

Create a lazy one-frame RGB or RGBA Signal with a shape-specific spec of the image modality.

Returns:

Type Description
Signal

A Signal whose single value is the original-size image tensor.

readers source

Validated readers for common source file formats.

CsvDataReader source

Bases: DataReader[RowT]

Read a UTF-8 CSV file with a header.

expected_columns instance-attribute source
expected_columns = expected_columns
path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated rows parsed from CSV cells.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or a row is invalid.

DataReader source

Bases: ABC, Generic[RowT]

Read and validate rows from one local source file.

The row schema determines validation strictness. CSV parses its string cells.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows abstractmethod source
iter_rows() -> Generator[RowT, None, None]

Yield validated rows in source order.

Close the iterator explicitly if stopping early to release its file.

Yields:

Type Description
RowT

Validated source rows.

DataRow source

Bases: BaseModel

A connector-owned schema for a source row.

model_config class-attribute instance-attribute source
model_config = ConfigDict(extra='forbid')

JsonArrayDataReader source

Bases: DataReader[RowT]

Read rows from a UTF-8 JSON array, loaded into memory.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated elements of the array.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or an element is invalid.

JsonlDataReader source

Bases: DataReader[RowT]

Read one JSON value per line of a UTF-8 file.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated lines.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or a line is invalid.

ParquetDataReader source

Bases: DataReader[RowT]

Read a Parquet file in batches, yielding individual rows.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
selected_columns instance-attribute source
selected_columns = (
    None
    if selected_columns is None
    else list(selected_columns)
)
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated rows in file order.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or a row is invalid.

base source

Common contract for validated source rows.

RowT module-attribute source
RowT = TypeVar('RowT', bound=DataRow)
DataReader source

Bases: ABC, Generic[RowT]

Read and validate rows from one local source file.

The row schema determines validation strictness. CSV parses its string cells.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows abstractmethod source
iter_rows() -> Generator[RowT, None, None]

Yield validated rows in source order.

Close the iterator explicitly if stopping early to release its file.

Yields:

Type Description
RowT

Validated source rows.
DataRow source

Bases: BaseModel

A connector-owned schema for a source row.

model_config class-attribute instance-attribute source
model_config = ConfigDict(extra='forbid')

csv source

Read CSV source rows.

CsvDataReader source

Bases: DataReader[RowT]

Read a UTF-8 CSV file with a header.

expected_columns instance-attribute source
expected_columns = expected_columns
path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated rows parsed from CSV cells.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or a row is invalid.

json source

Read JSON and JSON Lines source rows.

JsonArrayDataReader source

Bases: DataReader[RowT]

Read rows from a UTF-8 JSON array, loaded into memory.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated elements of the array.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or an element is invalid.
JsonlDataReader source

Bases: DataReader[RowT]

Read one JSON value per line of a UTF-8 file.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated lines.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or a line is invalid.

parquet source

Read Parquet source rows.

ParquetDataReader source

Bases: DataReader[RowT]

Read a Parquet file in batches, yielding individual rows.

path instance-attribute source
path = path
row_type instance-attribute source
row_type = row_type
selected_columns instance-attribute source
selected_columns = (
    None
    if selected_columns is None
    else list(selected_columns)
)
iter_rows source
iter_rows() -> Generator[RowT, None, None]

Yield validated rows in file order.

Yields:

Type Description
RowT

Validated source rows.

Raises:

Type Description
TimeNetConnectorError

If the file or a row is invalid.