-   [](https://platform.exaloop.io/)
-   Data and messages
-   Avro

# Avro

The Data SDK supports Avro container files and raw binary records. Read files as Arrow tables for columnar processing, or generate typed record classes when working with individual events. Both paths execute natively and support writing: tables use an explicit Avro schema, while generated records encode raw messages or write container files.

Choose table batches for analytical extraction and generated records for typed field access. The [API reference](https://platform.exaloop.io/data/avro-api/) documents schema resolution, logical types, and resource limits for both approaches.

**Before you begin:** install the [all-SDK distribution](https://platform.exaloop.io/start/distribution/). The [sample inputs](https://platform.exaloop.io/assets/files/avro-aa12f7dee74a690f2773894c75f4d7a4.zip/) contain the container files, raw records, and schemas used below. Extract the archive in your working directory before running the examples. Each input contains 100 trades with ID 1, so the batch example prints `100`. The generated-record example also shows its schema inline.

## Read Tables

```python title="program.codon"
import exaloop.data as data

with data.read_avro("events.avro") as events:
    identifiers = events["id"].to_numpy(int)
```

Run the example:

```sh title="Terminal"
exaloop --with data run -release program.codon
```

For an individual SDK, register it using the [installation guide](https://platform.exaloop.io/start/install/) or run `codon run -release -plugin /path/to/sdk/data program.codon` directly.

`data.read_table("events.avro")` infers Avro from the extension. Path-based readers and writers also accept `s3://bucket/key` and an optional `filesystem=data.S3FileSystem(...)`; see [S3 objects](https://platform.exaloop.io/data/data/#s3-objects) for credentials, object replacement, and upload failure handling. `data.Table.from_bytes(encoded, "avro")` reads an in-memory container file. Both materialize the entire table. To process bounded batches and select columns:

```python title="Source"
import exaloop.data.avro as avro

for batch in avro.iter_batches("events.avro", columns=["id"], batch_size=65536):
    with batch:
        identifiers = batch["id"].to_numpy(int)
        print(identifiers.sum())
```

This reads only `id` and prints a sum for each batch. Select the columns your application needs to avoid constructing unused values. The `with` block releases each batch after processing; a column you keep separately remains usable.

Projection skips unselected data, so it should not be used to validate every field in an input file.

## Generated Records

Generate records when you know the schema before compiling and want ordinary typed field access. Save this small schema as `trade.avsc`:

```json title="trade.avsc"
{
    "type": "record",
    "name": "Trade",
    "fields": [{"name": "id", "type": "long"}]
}
```

Generate the Codon module:

```sh title="Terminal"
exaloop avrogen trade.avsc trade.codon
```

The distribution registers the Data SDK automatically. Use `exaloop --with NAME avrogen ...` for another registered SDK alias. Without the CLI, run `/path/to/sdk/data/bin/avrogen-codon trade.avsc trade.codon`.

Place the generated module alongside your program:

```python title="Source"
from trade import Trade
from exaloop.data.avro import read_records

for record in read_records("trades.avro", Trade):
    print(record.id)
```

`Trade` is a generated record with a typed `id` field. Compile it together with your application; the deployed program does not need the generator.

For a raw binary message rather than a container file, use `Trade.from_bytes()`. When decoding many messages, prepare a decoder once and reuse it. For the schema above, the one-byte payload below represents a trade with ID 1:

```python title="Source"
from trade import Trade

decoder = Trade.decoder()
record = decoder.decode(b"\x02")
assert record.id == 1
```

If the incoming writer schema differs, pass it to `Trade.decoder(writer_schema)`. See the [generated-record reference](https://platform.exaloop.io/data/avro-api/#generated-records) for field types, unions, and naming rules.

## Construct Logical Values

Use logical-type helpers to construct timestamp, decimal, and UUID fields without calculating epoch counts or assembling bytes. Generated fields retain their physical Codon types; these helpers return values you can pass directly to them.

For example, generate `Payment` from this schema:

```json title="payment.avsc"
{
    "type": "record",
    "name": "Payment",
    "fields": [
        {"name": "created_at", "type": {"type": "long", "logicalType": "timestamp-micros"}},
        {"name": "amount", "type": {"type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 2}},
        {"name": "id", "type": {"type": "string", "logicalType": "uuid"}}
    ]
}
```

```sh title="Terminal"
exaloop avrogen payment.avsc payment.codon
```

```python title="Source"
from datetime import datetime
from payment import Payment
from exaloop.data.avro import timestamp, timestamp_datetime, decimal_bytes, decimal_string, uuid_string

payment = Payment(
        timestamp(datetime(2026, 10, 11, 9, 30), offset_minutes=330),
        decimal_bytes("123.45", precision=12, scale=2),
        uuid_string("00112233445566778899AABBCCDDEEFF"),
)
restored = Payment.from_bytes(payment.to_bytes())
assert timestamp_datetime(restored.created_at) == datetime(2026, 10, 11, 4, 0)
assert decimal_string(restored.amount, precision=12, scale=2) == "123.45"
assert restored.id == "00112233-4455-6677-8899-aabbccddeeff"
```

Here, `offset_minutes=330` means the supplied clock time is UTC+05:30. The default is UTC, never the machine's local timezone. Match `unit` to the schema: `"millis"`, `"micros"` (default), or `"nanos"`. For an Avro `local-timestamp-*`, use an offset of zero to retain the wall-clock value. Conversion rejects precision loss instead of rounding. Codon `datetime` has microsecond precision and no timezone metadata; use physical integer values directly when sub-microsecond timestamps are required.

Decimals accept exact decimal text, including scientific notation, or whole integers. Floats are not accepted. Match `precision` and `scale` to the schema; for a decimal stored as `fixed`, also pass its byte width as `fixed_size`. For a UUID stored as `fixed(16)`, use `uuid_bytes(text)` instead of `uuid_string(text)`. UUID helpers validate and normalize identifiers; they do not generate new IDs. See the [logical helper reference](https://platform.exaloop.io/data/avro-api/#logical-type-helpers) for limits and inverse conversions.

## Encode and Write Records

Use `to_bytes()` to encode a generated record as one raw Avro datum:

```python title="Source"
from trade import Trade

payload = Trade(42).to_bytes()
assert Trade.from_bytes(payload).id == 42
```

These bytes do not include a schema, container header, or schema-registry envelope. The receiving application needs the matching writer schema. Regenerate older modules with `exaloop avrogen` to add encoding methods; generated names such as a schema field named `to_bytes` may receive a `_field` suffix.

For a file, use `Writer[Record]`. It embeds the generated record's schema and writes bounded blocks as records arrive:

```python title="Source"
from trade import Trade
from exaloop.data.avro import Writer, read_records

with Writer[Trade]("written-trades.avro", codec="deflate") as output:
    for identifier in range(3):
        output.write(Trade(identifier))

assert [record.id for record in read_records("written-trades.avro", Trade)] == [0, 1, 2]
```

Opening a writer creates or truncates the destination. The context manager flushes pending records and closes it. Call `flush()` when another reader needs to see accepted records before the writer closes; this does not guarantee durability after a crash.

Writers support `null`, `deflate`, `snappy`, and `zstandard` compression without additional runtime dependencies. The default block target is 64 KiB, with at most 65,536 records per block and a 64 MiB hard limit on encoded records and compressed or uncompressed blocks. A record larger than the target can occupy its own block within that hard limit. These limits do not describe total process memory. See the [writer reference](https://platform.exaloop.io/data/avro-api/#container-writers) for configuration and failure behavior.

## Write Tables

Use `Table.write_avro()` to export an Arrow table without constructing generated records. Supply the writer schema explicitly; Arrow types alone do not identify Avro enums, unions, or named records unambiguously.

```python title="Source"
import exaloop.data as data
from exaloop.data.avro import Schema

schema = Schema('{"type":"record","name":"Trade","fields":[{"name":"id","type":"long"}]}')
with data.table({"id": data.array([10, 20, 30])}) as trades:
    trades.write_avro("table-trades.avro", schema, codec="deflate")

with data.read_avro("table-trades.avro") as restored:
    assert restored["id"].to_list(int) == [10, 20, 30]
```

The equivalent function is `avro.write_table(table, path, writer_schema, ...)`. Field names select columns and nested struct fields; schema order determines output order. Extra columns are ignored, but every schema field must exist, even if it has a default. Values are range-checked rather than silently truncated. See [table export types](https://platform.exaloop.io/data/avro-api/#table-writers) for nested and logical values.

Export creates or truncates the file and uses the same block limits and codecs as generated-record writing. Schema/layout errors are checked before opening the destination; a bad value later in the table can leave earlier records in the file. For atomic replacement, write a temporary file and rename it only after export succeeds. Calling `write_avro()` again replaces the file; it does not append another table or batch.

The input table stays in memory. Export retains its Arrow buffers and encodes records incrementally, but combining chunked columns can allocate additional memory. No Python runtime or generated record module is required.

## Runtime Schemas

Use a runtime schema when it arrives with the data or changes independently of your application. A writer schema describes the bytes being read; a reader schema describes the result your application expects.

This example reads 100 raw records, selecting their `id` field:

```python title="Source"
import exaloop.data.avro as avro

with open("writer.avsc") as stream:
    writer = avro.Schema(stream.read())
with open("reader.avsc") as stream:
    reader = avro.Schema(stream.read())
decoder = avro.Decoder(writer, reader, columns=["id"])
with open("records.bin", "rb") as stream:
    with decoder.decode(stream.read(), count=100) as records:
        print(records.num_rows)
```

Reuse the decoder for later batches with the same schemas. It handles compatible changes such as reordered fields, added reader fields with defaults, and numeric promotions. Keep both schemas unchanged while the decoder is in use.

## API

The [Avro API reference](https://platform.exaloop.io/data/avro-api/) lists reader and decoder signatures, schema evolution rules, and supported formats.

## Arrow Mapping

Table readers preserve Avro's structure and logical types in Arrow. For example, Avro `long` becomes Arrow int64 and a nullable field becomes a nullable column. See the [type mapping](https://platform.exaloop.io/data/avro-api/#arrow-mapping) when working with nested values, decimals, or timestamps.

## Limits and Ownership

Use batches when a file is too large to load entirely into memory. The default limits allow 64 MiB per compressed or decompressed block and 256 MiB of Arrow buffers per batch; these are not limits on total process memory. Keeping generated records can also retain their input blocks. Adjust limits only after considering the size and trustworthiness of your input.

Malformed or unsupported input raises `ValueError`. With an iterator, earlier batches may already have been processed before a later error occurs. The [limits reference](https://platform.exaloop.io/data/avro-api/#limits) covers configurable bounds and restrictions.

## Choosing a Reader

Tables are a natural fit for column-oriented analysis, while generated records provide typed access for event-by-event processing. The workload determines which is more efficient: representative files and the application's actual projection and decoding requirements provide a useful basis for comparison.

## Next Steps

-   Use the [generated-record reference](https://platform.exaloop.io/data/avro-api/#generated-records) for unions and nested fields.
-   Review [decoder limits](https://platform.exaloop.io/data/avro-api/#limits) before accepting external input.
-   [Consume Kafka records](https://platform.exaloop.io/streaming/kafka/) to apply decoding in a streaming application.

Source: https://platform.exaloop.io/data/avro/