Skip to content

Download

timenet_connectors.download source

Async, scheme-dispatching artifact downloads for connectors.

The high-level entry points are :func:download_files (a list of :class:Artifact, any mix of s3:// and http(s)://) and :func:ensure_archive (download a zip and extract it once). Downloads report progress through the ambient :func:progress_sink. The per-transport backends live in :mod:.http and :mod:.s3.

ProgressCallback module-attribute source

ProgressCallback = Callable[[DownloadProgress], None]

Artifact dataclass source

A file to download, with its URL, destination, and optional request settings and SHA-256 hash.

The headers and cookies apply only to HTTP downloads. The SHA-256 hash applies to both HTTP and S3 downloads.

cookies class-attribute instance-attribute source

cookies: Mapping[str, str] | None = None

dest instance-attribute source

dest: Path

headers class-attribute instance-attribute source

headers: Mapping[str, str] | None = None

sha256 class-attribute instance-attribute source

sha256: str | None = None

url instance-attribute source

url: str

DownloadProgress dataclass source

Progress for one download: bytes transferred so far and the total, if known.

downloaded instance-attribute source

downloaded: int

total instance-attribute source

total: int | None

url instance-attribute source

url: str

download_files async source

download_files(
    artifacts: Iterable[Artifact],
    *,
    headers: Mapping[str, str] | None = None,
    cookies: Mapping[str, str] | None = None,
    max_concurrency: int = _DEFAULT_MAX_CONCURRENCY,
    skip_existing: bool = True,
) -> list[Path]

Download files from S3 and HTTP URLs.

S3 downloads block the event loop and run one at a time, while HTTP downloads run concurrently. The function checks all URL schemes before starting any download.

Parameters:

Name Type Description Default
artifacts Iterable[Artifact]

The files to download, with their URLs and destination paths.

 required
headers Mapping[str, str] | None

Default HTTP headers that each artifact's headers override.

 None
cookies Mapping[str, str] | None

Default HTTP cookies that each artifact's cookies override.

 None
max_concurrency int

Maximum number of concurrent HTTP downloads.

 _DEFAULT_MAX_CONCURRENCY
skip_existing bool

Skip existing files if they match sha256 or have no expected hash.

 True

Returns:

Type Description
list[Path]

The destination paths, in input order.

Raises:

Type Description
TimeFValidationError

If a URL uses a scheme other than s3://, http://, or https://.

ensure_archive async source

ensure_archive(
    url: str,
    target: str | Path,
    *,
    filename: str | None = None,
    headers: Mapping[str, str] | None = None,
    cookies: Mapping[str, str] | None = None,
    sha256: str | None = None,
) -> Path

Download a zip archive and extract it into target.

A marker file lets later calls reuse an extraction with the same URL, filename, and SHA-256 hash. The function deletes the archive after extraction succeeds and keeps it if extraction fails.

Parameters:

Name Type Description Default
url str

The archive URL, s3:// or http(s)://.

 required
target str | Path

The directory for the archive and its extracted contents.

 required
filename str | None

An optional archive filename to use instead of the name derived from the URL.

 None
headers Mapping[str, str] | None

Request headers for HTTP downloads only.

 None
cookies Mapping[str, str] | None

Request cookies for HTTP downloads only.

 None
sha256 str | None

An optional SHA-256 hash in hexadecimal that the archive must match.

 None

Returns:

Type Description
Path

The extraction directory, target.

find_dir_containing source

find_dir_containing(root: Path, relative: str) -> Path

Find the directory under root that contains relative (archives extract nested).

Parameters:

Name Type Description Default
root Path

The extraction root to search.

 required
relative str

A file name expected inside the wanted directory.

 required

Returns:

Type Description
Path

The parent directory of the first match.

Raises:

Type Description
FileNotFoundError

If nothing matches.

progress_sink source

progress_sink(
    callback: ProgressCallback | None,
) -> Generator[None]

Install callback as the download-progress sink for the duration of the block.

Parameters:

Name Type Description Default
callback ProgressCallback | None

The sink to receive :class:DownloadProgress events, or None to report nothing.

 required

Yields:

Type Description
Generator[None]

Nothing. The sink is active within the with block and restored on exit.