Encoding Helpers

Internal module

http_to_arrow._encoding is an implementation helper module. Prefer the public ArrowRecordContainer API unless you are maintaining the package.

_encoding

Dictionary encoding helpers for ArrowRecordContainer.

These helpers decide when to dictionary-encode an Arrow array based on the caller's threshold, and produce an encoded array whose type stays stable across batches so pa.Table.from_batches accepts the batches.

is_dictionary_eligible_type(arrow_type)

Return True when arrow_type is a string variant we attempt to encode.

Source code in src/http_to_arrow/src/http_to_arrow/_encoding.py

def is_dictionary_eligible_type(arrow_type: pa.DataType) -> bool:
    """Return True when *arrow_type* is a string variant we attempt to encode."""
    return pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type)

maybe_dictionary_encode_array(array, logical_type, existing_effective_type, cardinality_threshold)

Optionally dictionary-encode array for low-cardinality string columns.

Behaviour rules:

• When the column has already been encoded in a prior batch (existing_effective_type is a dictionary type), the new array is always encoded so subsequent batches share the same physical type.
• When the logical column type is not a string/large-string, the array is returned unchanged.
• When the array is empty, no encoding decision can be made; return as-is.
• Otherwise the array is encoded and kept only if len(dictionary) / len(array) <= cardinality_threshold.

Source code in src/http_to_arrow/src/http_to_arrow/_encoding.py

def maybe_dictionary_encode_array(
    array: pa.Array,
    logical_type: pa.DataType,
    existing_effective_type: pa.DataType,
    cardinality_threshold: float,
) -> pa.Array:
    """Optionally dictionary-encode *array* for low-cardinality string columns.

    Behaviour rules:

    - When the column has already been encoded in a prior batch
      (``existing_effective_type`` is a dictionary type), the new array is
      always encoded so subsequent batches share the same physical type.
    - When the logical column type is not a string/large-string, the array
      is returned unchanged.
    - When the array is empty, no encoding decision can be made; return as-is.
    - Otherwise the array is encoded and kept only if
      ``len(dictionary) / len(array) <= cardinality_threshold``.
    """
    if pa.types.is_dictionary(existing_effective_type):
        return pc.dictionary_encode(array)

    if not is_dictionary_eligible_type(logical_type):
        return array

    if len(array) == 0:
        return array

    encoded = cast("pa.DictionaryArray", pc.dictionary_encode(array))
    if len(encoded.dictionary) / len(array) <= cardinality_threshold:
        return encoded

    return array