-   [](https://platform.exaloop.io/)
-   Data APIs
-   Avro API

# Avro API

Import readers, writers, schemas, and decoders from `exaloop.data.avro`. This reference defines their parameters, schema mappings, ownership rules, and limits. The [Avro guide](https://platform.exaloop.io/data/avro/) provides runnable examples for table processing and generated records.

Path-based readers and writers also accept an optional `filesystem`, defaulting to `None`. Omit it to use local paths or the default S3 configuration.

## Container Readers

```text title="Text"
read_table(path, reader_schema=None, columns=None, batch_size=65536,
           max_block_bytes=67108864, max_output_bytes=268435456) -> Table
read_bytes(data, reader_schema=None, columns=None, batch_size=65536,
           max_block_bytes=67108864, max_output_bytes=268435456) -> Table
iter_batches(path, reader_schema=None, columns=None, batch_size=65536,
             max_block_bytes=67108864, max_output_bytes=268435456)
read_records(path, Record: type, max_block_bytes=67108864)
```

Reader choice determines the output representation and materialization behavior: `read_table` and `read_bytes` load the entire container, `iter_batches` yields incremental tables, and `read_records` yields generated records.

| Parameter | Meaning |
| --- | --- |
| `reader_schema` | Optional `Schema` describing the desired result; defaults to the file's writer schema |
| `columns` | Top-level reader fields to select, in output order; `None` selects all |
| `batch_size` | Maximum records per returned table batch |
| `max_block_bytes` | Limit on each compressed and decompressed container block |
| `max_output_bytes` | Arrow buffer budget per batch, not a total memory limit |
| `filesystem` | Optional `data.S3FileSystem` for path-based readers; `s3://` also works with the default AWS client |

Table readers require a record schema at the root. Because a batch stays within one container block, it may contain fewer than `batch_size` rows. An empty file still yields one table carrying the schema.

## Container Writers

```text title="Text"
Writer[Record](path, codec="null", block_size=65536,
               batch_size=65536, max_block_bytes=67108864)
writer.write(record)
writer.flush()
writer.close()
writer.closed -> bool
```

`Record` must be a generated record with encoding support. The writer creates or truncates a local file, or replaces an S3 object on successful close, and embeds that record's schema. It supports context management; `close()` is idempotent. Appending to an existing container is not supported. Use the table-writing API below for Arrow tables.

| Parameter | Meaning |
| --- | --- |
| `codec` | `null`, `deflate`, `snappy`, or `zstandard` |
| `block_size` | Positive uncompressed block target, at most `max_block_bytes` |
| `batch_size` | Maximum records per block, from 1 through 1,048,576 |
| `max_block_bytes` | Hard limit on each encoded record and each compressed or uncompressed block; from 1 byte through 1 GiB |

A record larger than `block_size` is written in its own block if it fits the hard limit. Collection-work limits can cause earlier block boundaries. Compression can increase a block's size; exceeding either hard limit raises `ValueError`. Snappy's four-byte CRC32 trailer counts toward the compressed-block limit. Buffers are bounded and reused, but the limit is not a total memory budget.

`flush()` emits the pending block and flushes the file's buffers, without an `fsync` durability guarantee. Closing an unused writer produces a valid empty container. Closing also releases the writer's record and block buffers.

An invalid or oversized record raises `ValueError` before it is added to the pending block; the writer can accept another record afterward. A compression or file-output failure prevents further writing. Close that writer and treat the file as potentially partial. Filesystem errors propagate, and writes or flushes after close raise `ValueError`. A context manager closes even when its body raises; previously accepted records are flushed unless an output failure has occurred.

For S3, pass a `filesystem` or use an `s3://` URI. `flush()` does not publish the object. An exception escaping the writer context aborts the upload instead of committing previously accepted records; local-file exception behavior is unchanged. An invalid record caught inside the context still permits subsequent writes. Remote output errors trigger abort on close. Cleanup failure can leave an incomplete upload, and a lost completion response can leave a committed object despite an error. See the [S3 failure contract](https://platform.exaloop.io/data/data/#s3-objects).

## Table Writers

```text title="Text"
write_table(table, path, writer_schema: Schema, codec="null",
                  block_size=65536, batch_size=65536, max_block_bytes=67108864)
table.write_avro(path, writer_schema: Schema, codec="null",
                          block_size=65536, batch_size=65536, max_block_bytes=67108864)
```

Both APIs write a complete local container or S3 object and close it before returning. The schema must have a record root and must not be recursive. Field names match exactly, independently of column order; aliases and defaults are not applied. Missing fields and duplicate column/struct names are rejected; extra fields are ignored. Layout validation also runs for empty tables. Keep the source table, its borrowed storage, and the schema unchanged during export.

| Arrow input | Avro writer type / rules |
| --- | --- |
| Signed/unsigned integers | `int` or `long`, checked against the Avro signed range |
| float32 / float64 | `float` requires float32; `double` accepts either |
| Boolean / null | `boolean` / `null`; null values otherwise require a null union branch |
| String / large string | `string` or a declared enum symbol |
| Binary / large binary | `bytes` or exact-length `fixed` |
| Fixed-size binary | Matching-size `fixed` |
| Struct | Record; child fields selected by name |
| List / large list / fixed-size list | Array; child values validated recursively |
| Map | Map with non-null string keys; duplicate keys are rejected |
| Dictionary | Values decoded before applying these rules, including dictionary nulls |
| Dense/sparse union | Children must match schema branch order; Arrow type codes may differ |
| Nullable non-union column | Exactly one non-null Avro branch plus null; an Arrow null column can select null from any union |
| Decimal32/64/128/256 | Decimal bytes/fixed with matching scale; values checked against writer precision and fixed width |
| Date32 / time32(ms) / time64(us) | Matching Avro date / time-millis / time-micros |
| Timestamp(ms/us/ns) | Matching Avro timestamp unit; timezone-bearing arrays for timestamps, timezone-free arrays for local timestamps |

Integers and binary values can also supply logical types' physical storage. There is no automatic unit conversion, decimal rescaling, float-to-integer conversion, or inference of ambiguous unions. Avro timestamps preserve instants, not the original Arrow timezone label.

Codec, block, collection, and error limits match `Writer`. The input table and exported Arrow buffers remain resident, and combining column chunks can copy data. These allocations are outside the encoded-block budget.

Configuration and schema/layout errors do not truncate the destination. For local files, a value-dependent error excludes the failing record but can leave earlier valid records in the file; output failures can leave a partial block. For atomic replacement, write a temporary file and rename after successful completion. S3 errors instead use multipart abort with the failure caveats above. Generic `Table.write()` and `Table.to_bytes()` do not infer an Avro schema; use these explicit-schema file APIs instead.

## Raw Binary Decoders

```text title="Text"
Schema(source: str)
Decoder(writer_schema, reader_schema=None, columns=None,
        max_output_bytes=268435456)
decoder.decode(data: bytes, count: int = 1) -> Table
decode(data: bytes, writer_schema, reader_schema=None,
       count: int = 1, columns=None) -> Table
RecordDecoder[Record](writer_schema=None)
record_decoder.decode(data: bytes) -> Record
```

Construct `Schema` objects from JSON text and pass them to the reader or decoder. A reusable decoder avoids repeated setup for payloads sharing the same schemas; those schemas must remain unchanged for the decoder's lifetime.

Schema resolution supports reordered fields, reader aliases and defaults, omitted writer fields, numeric and string/bytes promotions, unions, and enum remapping with defaults. An incompatible selected union branch fails when read.

Records, enums, and fixed types match by their unqualified type names during resolution; namespaces still distinguish schema definitions and references. Reader aliases are resolved relative to the reader's namespace unless fully qualified. Reader unions select the first compatible branch, including permitted promotions, without preferring a later full-name or exact-type match. Other requirements, such as fixed sizes and decimal precision and scale, still apply.

All data-map readers use the last encoded value for a repeated key, including repetitions across collection blocks. Arrow output contains one entry per key; map entry order is not guaranteed. Overwritten entries still undergo normal validation and count toward collection and output limits. Arrow compaction buffers are charged to `max_output_bytes`; maps without duplicates do not require compaction. Caller-supplied Arrow maps must have unique keys when exported, and duplicate container metadata keys are rejected.

## Generated Records

`avrogen` produces typed `@tuple` classes. `Record.from_bytes(payload)` decodes one raw datum and rejects trailing bytes. Pass a writer `Schema` as its second argument to resolve a different input schema. `Record.decoder(writer_schema)` prepares a reusable decoder; omitting the schema uses the generated layout.

`record.to_bytes(max_bytes=67108864)` encodes one raw datum using the generated schema. It returns owned bytes without a schema header or registry envelope. `max_bytes` ranges from zero through 1 GiB; zero only permits an empty encoding. Every field is encoded, including values equal to schema defaults. Encoding checks fixed lengths, enum symbols, union branch/value agreement, collection limits, and output size. Regenerate existing modules to obtain these methods.

| Avro field | Generated Codon type |
| --- | --- |
| int / long | `i32` / `int` |
| float / double | `float32` / `float` |
| string / enum | `str` |
| bytes / fixed | `bytes` |
| array / map | Typed list / dictionary |
| Two-branch union with null | `Optional[T]` |
| Other union | Tagged value with `branch` and typed `value` |

A union's `branch` is its index in the generated schema: the reader schema when decoding, and the writer schema when encoding. Logical values retain their Avro storage type, such as integer timestamps and decimal bytes. Encoding does not implicitly convert logical objects; use the helpers below to prepare these values explicitly. Do not mutate the generated schema.

Reserved names receive a `_field` suffix and collisions receive trailing underscores. Inspect the generated class for the final spelling.

## Logical-Type Helpers

Import these functions from `exaloop.data.avro`. They return physical field values or their human-readable equivalents; generated record constructors and binary encoders remain unchanged. No Python runtime or additional dependency is needed.

| Function | Result |
| --- | --- |
| `timestamp(value: datetime, unit="micros", offset_minutes=0) -> int` | Exact signed epoch count for a timestamp field |
| `timestamp_datetime(value: int, unit="micros", offset_minutes=0) -> datetime` | Convert a physical timestamp to a Codon `datetime` |
| `decimal_bytes(value: str or int, precision, scale=0, fixed_size=0) -> bytes` | Exact unscaled decimal in Avro's signed big-endian representation |
| `decimal_string(value: bytes, precision, scale=0) -> str` | Decimal text, possibly in scientific notation |
| `uuid_string(value: str or bytes) -> str` | Canonical lowercase, hyphenated UUID text |
| `uuid_bytes(value: str) -> bytes` | Sixteen UUID bytes in network order, suitable for Avro `fixed(16)` |

**Timestamps.** Units are `millis`, `micros`, and `nanos`. The offset describes minutes east of UTC, from -1,439 through 1,439; encoding subtracts it and decoding adds it. Returned datetimes have no timezone metadata. No system timezone, daylight-saving rule, or timezone database is consulted. Local Avro timestamps use an offset of zero. A millisecond encoding requires an exact millisecond; a nanosecond decoding requires an exact microsecond. Overflow of Avro's signed 64-bit range or the datetime year range 1 through 9999 raises `ValueError`. No floating-point epoch arithmetic is used.

**Decimals.** Precision is 1 through 76 and scale is 0 through precision. Input text is at most 4,096 UTF-8 bytes and uses ASCII decimal syntax with an optional sign, decimal point, and exponent; whitespace, separators, NaN, and infinity are rejected. The input's parsed precision and normalized scale must also fit Decimal256's 76-digit limits. Rescaling can remove trailing zeros but never nonzero digits. Integer input denotes whole units, not an already-scaled coefficient. Negative zero becomes zero.

With `fixed_size=0`, encoding uses the shortest signed representation, including one byte for zero. A positive `fixed_size` specifies 1 byte through 1 MiB and must accommodate the declared precision, not just the current value; output is sign-padded to exactly that width. Decoding accepts up to 1 MiB of bytes, including redundant sign extension, and rejects empty or out-of-precision values. These helpers use Arrow's exact decimal arithmetic and raise `ValueError` on invalid values, precision loss, or unsupported limits.

**UUIDs.** Text must have 32 hexadecimal digits, optionally with the canonical 8-4-4-4-12 hyphen layout; either letter case is accepted. Binary input to `uuid_string` must contain exactly 16 bytes. URN prefixes, braces, and whitespace are not accepted. All bit patterns, including nil UUIDs, are preserved; version and variant bits are not rewritten. Invalid input raises `ValueError`.

## Arrow Mapping

| Avro type | Arrow output |
| --- | --- |
| int / long / float / double | int32 / int64 / float32 / float64 |
| boolean / null | Packed boolean / null |
| string / bytes | Large UTF-8 / large binary |
| fixed | Fixed-size binary |
| enum | Dictionary with int32 indices and string values |
| record / array / map | Struct / large list / map |
| `[null, T]` or `[T, null]` | Nullable `T` |
| Other unions | Dense tagged union, at most 128 branches |
| decimal | Decimal128 or Decimal256, up to 76 digits |
| date / time / timestamp | Arrow logical type; timestamps use UTC |
| local timestamp | Timestamp without timezone |
| UUID / duration | Underlying string or fixed binary |

Unknown or invalid logical annotations use their underlying Avro type. For nested or logical values without a native Data accessor, use Arrow IPC or the [C Data interface](https://platform.exaloop.io/data/native/#arrow-c-data).

## Limits

| Resource | Limit |
| --- | --- |
| Compressed or decompressed block | 64 MiB by default; configurable up to 1 GiB |
| Arrow buffers per batch | 256 MiB by default; set `max_output_bytes` to change |
| Schema text, encoded as UTF-8 | 1 MiB |
| Schema nodes / nesting depth | 8,192 / 64 |
| Resolution steps per plan | 65,536 |
| Resolution attempts per plan | 1,048,576 |
| Encoded reader defaults per plan | 8 MiB |
| Expanded default values per plan, including zero-byte values | 1,048,576 |
| Container header | 4 MiB |
| Expanded Arrow columns, including map keys | 8,192 |
| Records per block or items per collection | 1,048,576 |

Resolution and default limits apply when preparing a decoder, before reading records. Default budgets are shared across the plan, including repeated expansion of shared schema definitions, and are independent of `max_output_bytes`.

These are SDK limits, not Avro format limits. Further aggregate work limits apply to nested collections. Buffer budgets include enum dictionaries, but exclude other process memory and generated records. Decoders can cache dictionaries; retained records can keep their input blocks alive.

Readers and writers support `null`, `deflate`, `snappy`, and `zstandard`. DEFLATE writing uses compression level 6; Zstandard uses the bundled codec's default compression level. DEFLATE readers also accept the legacy Apache Python suffix only when its three bytes match the first three bytes of the uncompressed block's big-endian Adler-32 checksum. Other trailing bytes or mismatched suffixes are rejected. Writers always emit standard raw DEFLATE without a checksum suffix. Writer byte limits bound encoded records and blocks, not total process memory. Reusable record and block buffers grow geometrically; UTF-8 conversion and compression can allocate temporary buffers. Compression scratch buffers use the codec's worst-case size bound and can exceed `max_block_bytes` even when the result fits. Account for these allocations and retained input records when selecting limits for memory-constrained workers. Readers are single-threaded and accept local or S3 container files, or raw binary data. Writers are also single-threaded. JSON encoding, RPC, single-object fingerprints, other remote filesystems, append mode, and recursive generated records or table schemas are not supported.

These Data APIs read and write raw Avro or container files, not registry envelopes. Use the [Streaming registry client](https://platform.exaloop.io/streaming/kafka/#use-a-schema-registry) for Confluent-compatible framing, schema lookup, and generated Kafka record codecs.

Schema, framing, decoding, and limit failures raise `ValueError`; filesystem failures can raise `OSError`. Do not rely on exact exception text. An iterator may yield valid batches before encountering a malformed later block.

Source: https://platform.exaloop.io/data/avro-api/