Download

download_files(
    artifacts: Iterable[Artifact],
    *,
    headers: Mapping[str, str] | None = None,
    cookies: Mapping[str, str] | None = None,
    max_concurrency: int = _DEFAULT_MAX_CONCURRENCY,
    skip_existing: bool = True,
) -> list[Path]

Download a list of artifacts, mixing s3:// and http(s):// freely.

HTTP entries download concurrently (bounded by max_concurrency, sharing one connection pool). S3 entries download one at a time, since boto3 blocks the event loop but parallelizes each transfer itself. Batch headers and cookies apply to every HTTP request, under each artifact's own (ignored for S3). The function validates every scheme up front, so an unsupported one fails before any download starts.

Parameters:

Name	Type	Description	Default
artifacts	Iterable[Artifact]	The artifacts to download. A single file is a one-element list.	required
headers	Mapping[str, str] | None	Headers applied to every HTTP request, under each artifact's own.	None
cookies	Mapping[str, str] | None	Cookies applied to every HTTP request, under each artifact's own.	None
max_concurrency	int	Maximum number of concurrent HTTP downloads.	_DEFAULT_MAX_CONCURRENCY
skip_existing	bool	Skip any artifact whose destination already exists.	True

Returns:

Type	Description
list[Path]	The destination paths, in input order.

Raises:

Type	Description
TimeFValidationError	If any URL is neither an s3:// nor an http(s):// URL.

ensure_archive(
    url: str,
    target: str | Path,
    *,
    filename: str | None = None,
    headers: Mapping[str, str] | None = None,
    cookies: Mapping[str, str] | None = None,
    sha256: str | None = None,
) -> Path

Download a zip archive (s3:// or http(s)://) and extract it into target once.

Idempotent: the function writes a marker file under target, keyed by the archive URL, after a successful extraction. A re-run then reuses the extracted contents and skips the download.

A successful extraction then deletes the archive, because the marker alone makes the next run cheap. A failed one keeps it, so the next run extracts it again instead of downloading it again.

Parameters:

Name	Type	Description	Default
url	str	The archive URL, s3:// or http(s)://.	required
target	str | Path	Directory into which the function downloads the archive and extracts its contents.	required
filename	str | None	Overrides the cached archive name, for URLs whose path has no usable filename (a trailing slash or a /download suffix).	None
headers	Mapping[str, str] | None	Request headers for an HTTP download (ignored for S3).	None
cookies	Mapping[str, str] | None	Request cookies for an HTTP download (ignored for S3).	None
sha256	str | None	Optional hex digest the downloaded archive must match (HTTP only).	None

Returns:

Type	Description
Path	target (the extraction root).

find_dir_containing(root: Path, relative: str) -> Path

Find the directory under root that contains relative (archives extract nested).

Parameters:

Name	Type	Description	Default
root	Path	The extraction root to search.	required
relative	str	A file name expected inside the wanted directory.	required

Returns:

Type	Description
Path	The parent directory of the first match.

Raises:

Type	Description
FileNotFoundError	If nothing matches.

progress_sink(
    callback: ProgressCallback | None,
) -> Iterator[None]

Install callback as the download-progress sink for the duration of the block.

Parameters:

Name	Type	Description	Default
callback	ProgressCallback | None	The sink to receive :class:DownloadProgress events, or None to report nothing.	required

Yields:

Type	Description
None	Nothing. The sink is active within the with block and restored on exit.