Avro API
Import readers, writers, schemas, and decoders from exaloop.data.avro.
This reference defines their parameters, schema mappings, ownership rules, and
limits. The Avro guide provides runnable examples for table processing
and generated records.
Path-based readers and writers also accept an optional filesystem, defaulting
to None. Omit it to use local paths or the default S3 configuration.
Container Readers
read_table(path, reader_schema=None, columns=None, batch_size=65536,
max_block_bytes=67108864, max_output_bytes=268435456) -> Table
read_bytes(data, reader_schema=None, columns=None, batch_size=65536,
max_block_bytes=67108864, max_output_bytes=268435456) -> Table
iter_batches(path, reader_schema=None, columns=None, batch_size=65536,
max_block_bytes=67108864, max_output_bytes=268435456)
read_records(path, Record: type, max_block_bytes=67108864)
Reader choice determines the output representation and materialization behavior:
read_table and read_bytes load the entire container, iter_batches yields
incremental tables, and read_records yields generated records.
| Parameter | Meaning |
|---|---|
reader_schema | Optional Schema describing the desired result; defaults to the file's writer schema |
columns | Top-level reader fields to select, in output order; None selects all |
batch_size | Maximum records per returned table batch |
max_block_bytes | Limit on each compressed and decompressed container block |
max_output_bytes | Arrow buffer budget per batch, not a total memory limit |
filesystem | Optional data.S3FileSystem for path-based readers; s3:// also works with the default AWS client |
Table readers require a record schema at the root. Because a batch stays within
one container block, it may contain fewer than batch_size rows. An empty file
still yields one table carrying the schema.
Container Writers
Writer[Record](path, codec="null", block_size=65536,
batch_size=65536, max_block_bytes=67108864)
writer.write(record)
writer.flush()
writer.close()
writer.closed -> bool
Record must be a generated record with encoding support. The writer creates
or truncates a local file, or replaces an S3 object on successful close, and embeds
that record's schema. It supports context
management; close() is idempotent. Appending to an existing container is not
supported. Use the table-writing API below for Arrow tables.
| Parameter | Meaning |
|---|---|
codec | null, deflate, snappy, or zstandard |
block_size | Positive uncompressed block target, at most max_block_bytes |
batch_size | Maximum records per block, from 1 through 1,048,576 |
max_block_bytes | Hard limit on each encoded record and each compressed or uncompressed block; from 1 byte through 1 GiB |
A record larger than block_size is written in its own block if it fits the
hard limit. Collection-work limits can cause earlier block boundaries. Compression
can increase a block's size; exceeding either hard limit raises ValueError.
Snappy's four-byte CRC32 trailer counts toward the compressed-block limit.
Buffers are bounded and reused, but the limit is not a total memory budget.
flush() emits the pending block and flushes the file's buffers, without an
fsync durability guarantee. Closing an unused writer produces a valid empty
container. Closing also releases the writer's record and block buffers.
An invalid or oversized record raises ValueError before it is added to the
pending block; the writer can accept another record afterward. A compression or
file-output failure prevents further writing. Close that writer and treat the
file as potentially partial. Filesystem errors propagate, and writes or flushes
after close raise ValueError. A context manager closes even when its body raises;
previously accepted records are flushed unless an output failure has occurred.
For S3, pass a filesystem or use an s3:// URI. flush() does not publish the
object. An exception escaping the writer context aborts the upload instead of
committing previously accepted records; local-file exception behavior is
unchanged. An invalid record caught inside the context still permits subsequent
writes. Remote output errors trigger abort on close. Cleanup failure can leave
an incomplete upload, and a lost completion response can leave a committed object
despite an error. See the S3 failure contract.
Table Writers
write_table(table, path, writer_schema: Schema, codec="null",
block_size=65536, batch_size=65536, max_block_bytes=67108864)
table.write_avro(path, writer_schema: Schema, codec="null",
block_size=65536, batch_size=65536, max_block_bytes=67108864)
Both APIs write a complete local container or S3 object and close it before returning. The schema must have a record root and must not be recursive. Field names match exactly, independently of column order; aliases and defaults are not applied. Missing fields and duplicate column/struct names are rejected; extra fields are ignored. Layout validation also runs for empty tables. Keep the source table, its borrowed storage, and the schema unchanged during export.
| Arrow input | Avro writer type / rules |
|---|---|
| Signed/unsigned integers | int or long, checked against the Avro signed range |
| float32 / float64 | float requires float32; double accepts either |
| Boolean / null | boolean / null; null values otherwise require a null union branch |
| String / large string | string or a declared enum symbol |
| Binary / large binary | bytes or exact-length fixed |
| Fixed-size binary | Matching-size fixed |
| Struct | Record; child fields selected by name |
| List / large list / fixed-size list | Array; child values validated recursively |
| Map | Map with non-null string keys; duplicate keys are rejected |
| Dictionary | Values decoded before applying these rules, including dictionary nulls |
| Dense/sparse union | Children must match schema branch order; Arrow type codes may differ |
| Nullable non-union column | Exactly one non-null Avro branch plus null; an Arrow null column can select null from any union |
| Decimal32/64/128/256 | Decimal bytes/fixed with matching scale; values checked against writer precision and fixed width |
| Date32 / time32(ms) / time64(us) | Matching Avro date / time-millis / time-micros |
| Timestamp(ms/us/ns) | Matching Avro timestamp unit; timezone-bearing arrays for timestamps, timezone-free arrays for local timestamps |
Integers and binary values can also supply logical types' physical storage. There is no automatic unit conversion, decimal rescaling, float-to-integer conversion, or inference of ambiguous unions. Avro timestamps preserve instants, not the original Arrow timezone label.
Codec, block, collection, and error limits match Writer. The input table and
exported Arrow buffers remain resident, and combining column chunks can copy
data. These allocations are outside the encoded-block budget.
Configuration and schema/layout errors do not truncate the destination. For local files, a
value-dependent error excludes the failing record but can leave earlier valid
records in the file; output failures can leave a partial block. For atomic
replacement, write a temporary file and rename after successful completion.
S3 errors instead use multipart abort with the failure caveats above.
Generic Table.write() and Table.to_bytes() do not infer an Avro schema;
use these explicit-schema file APIs instead.
Raw Binary Decoders
Schema(source: str)
Decoder(writer_schema, reader_schema=None, columns=None,
max_output_bytes=268435456)
decoder.decode(data: bytes, count: int = 1) -> Table
decode(data: bytes, writer_schema, reader_schema=None,
count: int = 1, columns=None) -> Table
RecordDecoder[Record](writer_schema=None)
record_decoder.decode(data: bytes) -> Record
Construct Schema objects from JSON text and pass them to the reader or decoder.
A reusable decoder avoids repeated setup for payloads sharing the same schemas;
those schemas must remain unchanged for the decoder's lifetime.
Schema resolution supports reordered fields, reader aliases and defaults, omitted writer fields, numeric and string/bytes promotions, unions, and enum remapping with defaults. An incompatible selected union branch fails when read.
Records, enums, and fixed types match by their unqualified type names during resolution; namespaces still distinguish schema definitions and references. Reader aliases are resolved relative to the reader's namespace unless fully qualified. Reader unions select the first compatible branch, including permitted promotions, without preferring a later full-name or exact-type match. Other requirements, such as fixed sizes and decimal precision and scale, still apply.
All data-map readers use the last encoded value for a repeated key, including
repetitions across collection blocks. Arrow output contains one entry per key;
map entry order is not guaranteed. Overwritten entries still undergo normal
validation and count toward collection and output limits. Arrow compaction
buffers are charged to max_output_bytes; maps without duplicates do not require
compaction. Caller-supplied Arrow maps must have unique keys when exported, and
duplicate container metadata keys are rejected.
Generated Records
avrogen produces typed @tuple classes. Record.from_bytes(payload) decodes
one raw datum and rejects trailing bytes. Pass a writer Schema as its second
argument to resolve a different input schema. Record.decoder(writer_schema)
prepares a reusable decoder; omitting the schema uses the generated layout.
record.to_bytes(max_bytes=67108864) encodes one raw datum using the generated
schema. It returns owned bytes without a schema header or registry envelope.
max_bytes ranges from zero through 1 GiB; zero only permits an empty encoding.
Every field is encoded, including values equal to schema defaults. Encoding
checks fixed lengths, enum symbols, union branch/value agreement, collection
limits, and output size. Regenerate existing modules to obtain these methods.
| Avro field | Generated Codon type |
|---|---|
| int / long | i32 / int |
| float / double | float32 / float |
| string / enum | str |
| bytes / fixed | bytes |
| array / map | Typed list / dictionary |
| Two-branch union with null | Optional[T] |
| Other union | Tagged value with branch and typed value |
A union's branch is its index in the generated schema: the reader schema when
decoding, and the writer schema when encoding. Logical values retain their Avro
storage type, such as integer timestamps and decimal bytes. Encoding does not
implicitly convert logical objects; use the helpers below to prepare these
values explicitly. Do not mutate the generated schema.
Reserved names receive a _field suffix and collisions receive trailing
underscores. Inspect the generated class for the final spelling.
Logical-Type Helpers
Import these functions from exaloop.data.avro. They return physical field
values or their human-readable equivalents; generated record constructors and
binary encoders remain unchanged. No Python runtime or additional dependency
is needed.
| Function | Result |
|---|---|
timestamp(value: datetime, unit="micros", offset_minutes=0) -> int | Exact signed epoch count for a timestamp field |
timestamp_datetime(value: int, unit="micros", offset_minutes=0) -> datetime | Convert a physical timestamp to a Codon datetime |
decimal_bytes(value: str or int, precision, scale=0, fixed_size=0) -> bytes | Exact unscaled decimal in Avro's signed big-endian representation |
decimal_string(value: bytes, precision, scale=0) -> str | Decimal text, possibly in scientific notation |
uuid_string(value: str or bytes) -> str | Canonical lowercase, hyphenated UUID text |
uuid_bytes(value: str) -> bytes | Sixteen UUID bytes in network order, suitable for Avro fixed(16) |
Timestamps. Units are millis, micros, and nanos. The offset describes
minutes east of UTC, from -1,439 through 1,439; encoding subtracts it and
decoding adds it. Returned datetimes have no timezone metadata. No system
timezone, daylight-saving rule, or timezone database is consulted. Local Avro
timestamps use an offset of zero. A millisecond encoding requires an exact
millisecond; a nanosecond decoding requires an exact microsecond. Overflow of
Avro's signed 64-bit range or the datetime year range 1 through 9999 raises
ValueError. No floating-point epoch arithmetic is used.
Decimals. Precision is 1 through 76 and scale is 0 through precision. Input text is at most 4,096 UTF-8 bytes and uses ASCII decimal syntax with an optional sign, decimal point, and exponent; whitespace, separators, NaN, and infinity are rejected. The input's parsed precision and normalized scale must also fit Decimal256's 76-digit limits. Rescaling can remove trailing zeros but never nonzero digits. Integer input denotes whole units, not an already-scaled coefficient. Negative zero becomes zero.
With fixed_size=0, encoding uses the shortest signed representation, including
one byte for zero. A positive fixed_size specifies 1 byte through 1 MiB and
must accommodate the declared precision, not just the current value; output is
sign-padded to exactly that width. Decoding accepts up to 1 MiB of bytes, including
redundant sign extension, and rejects empty or out-of-precision values. These
helpers use Arrow's exact decimal arithmetic and raise ValueError on invalid
values, precision loss, or unsupported limits.
UUIDs. Text must have 32 hexadecimal digits, optionally with the canonical
8-4-4-4-12 hyphen layout; either letter case is accepted. Binary input to
uuid_string must contain exactly 16 bytes. URN prefixes, braces, and whitespace
are not accepted. All bit patterns, including nil UUIDs, are preserved; version
and variant bits are not rewritten. Invalid input raises ValueError.
Arrow Mapping
| Avro type | Arrow output |
|---|---|
| int / long / float / double | int32 / int64 / float32 / float64 |
| boolean / null | Packed boolean / null |
| string / bytes | Large UTF-8 / large binary |
| fixed | Fixed-size binary |
| enum | Dictionary with int32 indices and string values |
| record / array / map | Struct / large list / map |
[null, T] or [T, null] | Nullable T |
| Other unions | Dense tagged union, at most 128 branches |
| decimal | Decimal128 or Decimal256, up to 76 digits |
| date / time / timestamp | Arrow logical type; timestamps use UTC |
| local timestamp | Timestamp without timezone |
| UUID / duration | Underlying string or fixed binary |
Unknown or invalid logical annotations use their underlying Avro type. For nested or logical values without a native Data accessor, use Arrow IPC or the C Data interface.
Limits
| Resource | Limit |
|---|---|
| Compressed or decompressed block | 64 MiB by default; configurable up to 1 GiB |
| Arrow buffers per batch | 256 MiB by default; set max_output_bytes to change |
| Schema text, encoded as UTF-8 | 1 MiB |
| Schema nodes / nesting depth | 8,192 / 64 |
| Resolution steps per plan | 65,536 |
| Resolution attempts per plan | 1,048,576 |
| Encoded reader defaults per plan | 8 MiB |
| Expanded default values per plan, including zero-byte values | 1,048,576 |
| Container header | 4 MiB |
| Expanded Arrow columns, including map keys | 8,192 |
| Records per block or items per collection | 1,048,576 |
Resolution and default limits apply when preparing a decoder, before reading
records. Default budgets are shared across the plan, including repeated expansion
of shared schema definitions, and are independent of max_output_bytes.
These are SDK limits, not Avro format limits. Further aggregate work limits apply to nested collections. Buffer budgets include enum dictionaries, but exclude other process memory and generated records. Decoders can cache dictionaries; retained records can keep their input blocks alive.
Readers and writers support null, deflate, snappy, and zstandard.
DEFLATE writing uses compression level 6; Zstandard uses the bundled codec's
default compression level.
DEFLATE readers also accept the legacy Apache Python suffix only when its three
bytes match the first three bytes of the uncompressed block's big-endian Adler-32
checksum. Other trailing bytes or mismatched suffixes are rejected. Writers always
emit standard raw DEFLATE without a checksum suffix.
Writer byte limits bound encoded records and blocks, not total process memory.
Reusable record and block buffers grow geometrically; UTF-8 conversion and
compression can allocate temporary buffers. Compression scratch buffers use the
codec's worst-case size bound and can exceed max_block_bytes even when the
result fits. Account for these allocations and
retained input records when selecting limits for memory-constrained workers.
Readers are single-threaded and accept local or S3 container files, or raw binary data.
Writers are also single-threaded. JSON encoding, RPC,
single-object fingerprints, other remote filesystems,
append mode, and recursive generated records or table schemas are not supported.
These Data APIs read and write raw Avro or container files, not registry envelopes. Use the Streaming registry client for Confluent-compatible framing, schema lookup, and generated Kafka record codecs.
Schema, framing, decoding, and limit failures raise ValueError; filesystem
failures can raise OSError. Do not rely on exact exception text. An iterator
may yield valid batches before encountering a malformed later block.