`ViaFiles`¶

ViaFiles is the async downloader used for Defender file-based export workflows.

Public exports¶

ViaFiles
ViaFilesConfig
EmptyExportBlobError

What `ViaFiles` does¶

Given a set of export URLs, ViaFiles:

downloads blobs concurrently
detects gzip or plain text responses
parses newline-delimited JSON incrementally
batches records into an ArrowRecordContainer
retries transient failures

Most package users do not need to call ViaFiles directly because export-backed endpoint wrappers already use it internally.

Streaming into an async sink¶

stream_export_files(urls, sink, *, record_transform=None) mirrors download_export_files, but instead of accumulating an ArrowRecordContainer it feeds each parsed batch into any async sink exposing async def extend(rows) — for example an http_to_arrow.ArrowIPCStream. This is what BaseResults.to_ipc_stream() uses to stream export-backed endpoints without materializing the full dataset.

`ViaFilesConfig`¶

ViaFilesConfig exposes these tuning fields:

download_workers
client_timeout
retry_attempts
retry_delay_seconds
download_chunk_size
parse_batch_size

These settings control concurrency, timeout behavior, retry policy, and streaming batch sizes.

Error behavior¶

EmptyExportBlobError is raised internally when a blob returns 200 OK but contains no records. After retries, confirmed-empty blobs are skipped instead of aborting the whole download.

Repeated hard failures raise a runtime error.

API¶

viaFiles ¶

Async streaming downloader for Defender file-based export endpoints.

When an MDE export endpoint returns a list of SAS-signed blob URLs (exportFiles), this module downloads them concurrently, decompresses gzip on the fly, parses newline-delimited JSON, and streams the parsed records into an ArrowRecordContainer in batches.

The public entry point is :pymethod:ViaFiles.download_export_files.

ViaFiles ¶

ViaFiles(config: ViaFilesConfig | None = None)

Async streaming downloader for Defender file-based exports.

Accepts an optional :class:ViaFilesConfig to control concurrency, timeouts, retry behaviour, and batch sizes. When none is supplied the defaults from ViaFilesConfig are used.

download_export_files `async` ¶

download_export_files(
    urls: list[str],
    container: ArrowRecordContainer,
    *,
    record_transform: RecordTransform | None = None,
) -> ArrowRecordContainer

Download export blobs concurrently and stream records into container.

Parameters:

Name	Type	Description	Default
`urls`	`list[str]`	SAS-signed blob URLs returned by an MDE `exportFiles` response.	required
`container`	`ArrowRecordContainer`	Pre-configured :class:`ArrowRecordContainer` that records are streamed into.	required
`record_transform`	`RecordTransform \| None`	Optional callable applied to each parsed JSON record before it is batched. Use this to flatten nested export formats (e.g. `DeviceGatheredInfo`).	`None`

Returns:

Type	Description
`ArrowRecordContainer`	The same container, now populated with the parsed records.

stream_export_files `async` ¶

stream_export_files(
    urls: list[str],
    sink: AsyncRecordSink,
    *,
    record_transform: RecordTransform | None = None,
) -> None

Download export blobs concurrently and stream records into sink.

Like :meth:download_export_files, but instead of accumulating an :class:ArrowRecordContainer, each parsed batch is fed into sink via await sink.extend(...). This pipes export records straight into an :class:~http_to_arrow.ArrowIPCStream without materializing the full dataset, keeping peak memory close to a single batch.

Parameters:

Name	Type	Description	Default
`urls`	`list[str]`	SAS-signed blob URLs returned by an MDE `exportFiles` response.	required
`sink`	`AsyncRecordSink`	An object exposing `async def extend(rows)` (for example an `ArrowIPCStream`) that consumes each parsed batch.	required
`record_transform`	`RecordTransform \| None`	Optional callable applied to each parsed JSON record before it is batched.	`None`

ViaFilesConfig ¶

Bases: BaseModel

Tuning knobs for :class:ViaFiles export downloads.

EmptyExportBlobError ¶

Bases: RuntimeError

Raised when an export blob returns 200 OK but contains no records.

ViaFiles¶