Skip to main content
Markdown

Avro

The Data SDK supports Avro container files and raw binary records. Read files as Arrow tables for columnar processing, or generate typed record classes when working with individual events. Both paths execute natively and support writing: tables use an explicit Avro schema, while generated records encode raw messages or write container files.

Choose table batches for analytical extraction and generated records for typed field access. The API reference documents schema resolution, logical types, and resource limits for both approaches.

Before you begin: install the all-SDK distribution. The sample inputs contain the container files, raw records, and schemas used below. Extract the archive in your working directory before running the examples. Each input contains 100 trades with ID 1, so the batch example prints 100. The generated-record example also shows its schema inline.

Read Tables​

program.codon
import exaloop.data as data

with data.read_avro("events.avro") as events:
    identifiers = events["id"].to_numpy(int)

Run the example:

Terminal
exaloop --with data run -release program.codon

For an individual SDK, register it using the installation guide or run codon run -release -plugin /path/to/sdk/data program.codon directly.

data.read_table("events.avro") infers Avro from the extension. Path-based readers and writers also accept s3://bucket/key and an optional filesystem=data.S3FileSystem(...); see S3 objects for credentials, object replacement, and upload failure handling. data.Table.from_bytes(encoded, "avro") reads an in-memory container file. Both materialize the entire table. To process bounded batches and select columns:

Source
import exaloop.data.avro as avro

for batch in avro.iter_batches("events.avro", columns=["id"], batch_size=65536):
    with batch:
        identifiers = batch["id"].to_numpy(int)
        print(identifiers.sum())

This reads only id and prints a sum for each batch. Select the columns your application needs to avoid constructing unused values. The with block releases each batch after processing; a column you keep separately remains usable.

Projection skips unselected data, so it should not be used to validate every field in an input file.

Generated Records​

Generate records when you know the schema before compiling and want ordinary typed field access. Save this small schema as trade.avsc:

trade.avsc
{
    "type": "record",
    "name": "Trade",
    "fields": [{"name": "id", "type": "long"}]
}

Generate the Codon module:

Terminal
exaloop avrogen trade.avsc trade.codon

The distribution registers the Data SDK automatically. Use exaloop --with NAME avrogen ... for another registered SDK alias. Without the CLI, run /path/to/sdk/data/bin/avrogen-codon trade.avsc trade.codon.

Place the generated module alongside your program:

Source
from trade import Trade
from exaloop.data.avro import read_records

for record in read_records("trades.avro", Trade):
    print(record.id)

Trade is a generated record with a typed id field. Compile it together with your application; the deployed program does not need the generator.

For a raw binary message rather than a container file, use Trade.from_bytes(). When decoding many messages, prepare a decoder once and reuse it. For the schema above, the one-byte payload below represents a trade with ID 1:

Source
from trade import Trade

decoder = Trade.decoder()
record = decoder.decode(b"\x02")
assert record.id == 1

If the incoming writer schema differs, pass it to Trade.decoder(writer_schema). See the generated-record reference for field types, unions, and naming rules.

Construct Logical Values​

Use logical-type helpers to construct timestamp, decimal, and UUID fields without calculating epoch counts or assembling bytes. Generated fields retain their physical Codon types; these helpers return values you can pass directly to them.

For example, generate Payment from this schema:

payment.avsc
{
    "type": "record",
    "name": "Payment",
    "fields": [
        {"name": "created_at", "type": {"type": "long", "logicalType": "timestamp-micros"}},
        {"name": "amount", "type": {"type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 2}},
        {"name": "id", "type": {"type": "string", "logicalType": "uuid"}}
    ]
}
Terminal
exaloop avrogen payment.avsc payment.codon
Source
from datetime import datetime
from payment import Payment
from exaloop.data.avro import timestamp, timestamp_datetime, decimal_bytes, decimal_string, uuid_string

payment = Payment(
        timestamp(datetime(2026, 10, 11, 9, 30), offset_minutes=330),
        decimal_bytes("123.45", precision=12, scale=2),
        uuid_string("00112233445566778899AABBCCDDEEFF"),
)
restored = Payment.from_bytes(payment.to_bytes())
assert timestamp_datetime(restored.created_at) == datetime(2026, 10, 11, 4, 0)
assert decimal_string(restored.amount, precision=12, scale=2) == "123.45"
assert restored.id == "00112233-4455-6677-8899-aabbccddeeff"

Here, offset_minutes=330 means the supplied clock time is UTC+05:30. The default is UTC, never the machine's local timezone. Match unit to the schema: "millis", "micros" (default), or "nanos". For an Avro local-timestamp-*, use an offset of zero to retain the wall-clock value. Conversion rejects precision loss instead of rounding. Codon datetime has microsecond precision and no timezone metadata; use physical integer values directly when sub-microsecond timestamps are required.

Decimals accept exact decimal text, including scientific notation, or whole integers. Floats are not accepted. Match precision and scale to the schema; for a decimal stored as fixed, also pass its byte width as fixed_size. For a UUID stored as fixed(16), use uuid_bytes(text) instead of uuid_string(text). UUID helpers validate and normalize identifiers; they do not generate new IDs. See the logical helper reference for limits and inverse conversions.

Encode and Write Records​

Use to_bytes() to encode a generated record as one raw Avro datum:

Source
from trade import Trade

payload = Trade(42).to_bytes()
assert Trade.from_bytes(payload).id == 42

These bytes do not include a schema, container header, or schema-registry envelope. The receiving application needs the matching writer schema. Regenerate older modules with exaloop avrogen to add encoding methods; generated names such as a schema field named to_bytes may receive a _field suffix.

For a file, use Writer[Record]. It embeds the generated record's schema and writes bounded blocks as records arrive:

Source
from trade import Trade
from exaloop.data.avro import Writer, read_records

with Writer[Trade]("written-trades.avro", codec="deflate") as output:
    for identifier in range(3):
        output.write(Trade(identifier))

assert [record.id for record in read_records("written-trades.avro", Trade)] == [0, 1, 2]

Opening a writer creates or truncates the destination. The context manager flushes pending records and closes it. Call flush() when another reader needs to see accepted records before the writer closes; this does not guarantee durability after a crash.

Writers support null, deflate, snappy, and zstandard compression without additional runtime dependencies. The default block target is 64 KiB, with at most 65,536 records per block and a 64 MiB hard limit on encoded records and compressed or uncompressed blocks. A record larger than the target can occupy its own block within that hard limit. These limits do not describe total process memory. See the writer reference for configuration and failure behavior.

Write Tables​

Use Table.write_avro() to export an Arrow table without constructing generated records. Supply the writer schema explicitly; Arrow types alone do not identify Avro enums, unions, or named records unambiguously.

Source
import exaloop.data as data
from exaloop.data.avro import Schema

schema = Schema('{"type":"record","name":"Trade","fields":[{"name":"id","type":"long"}]}')
with data.table({"id": data.array([10, 20, 30])}) as trades:
    trades.write_avro("table-trades.avro", schema, codec="deflate")

with data.read_avro("table-trades.avro") as restored:
    assert restored["id"].to_list(int) == [10, 20, 30]

The equivalent function is avro.write_table(table, path, writer_schema, ...). Field names select columns and nested struct fields; schema order determines output order. Extra columns are ignored, but every schema field must exist, even if it has a default. Values are range-checked rather than silently truncated. See table export types for nested and logical values.

Export creates or truncates the file and uses the same block limits and codecs as generated-record writing. Schema/layout errors are checked before opening the destination; a bad value later in the table can leave earlier records in the file. For atomic replacement, write a temporary file and rename it only after export succeeds. Calling write_avro() again replaces the file; it does not append another table or batch.

The input table stays in memory. Export retains its Arrow buffers and encodes records incrementally, but combining chunked columns can allocate additional memory. No Python runtime or generated record module is required.

Runtime Schemas​

Use a runtime schema when it arrives with the data or changes independently of your application. A writer schema describes the bytes being read; a reader schema describes the result your application expects.

This example reads 100 raw records, selecting their id field:

Source
import exaloop.data.avro as avro

with open("writer.avsc") as stream:
    writer = avro.Schema(stream.read())
with open("reader.avsc") as stream:
    reader = avro.Schema(stream.read())
decoder = avro.Decoder(writer, reader, columns=["id"])
with open("records.bin", "rb") as stream:
    with decoder.decode(stream.read(), count=100) as records:
        print(records.num_rows)

Reuse the decoder for later batches with the same schemas. It handles compatible changes such as reordered fields, added reader fields with defaults, and numeric promotions. Keep both schemas unchanged while the decoder is in use.

API​

The Avro API reference lists reader and decoder signatures, schema evolution rules, and supported formats.

Arrow Mapping​

Table readers preserve Avro's structure and logical types in Arrow. For example, Avro long becomes Arrow int64 and a nullable field becomes a nullable column. See the type mapping when working with nested values, decimals, or timestamps.

Limits and Ownership​

Use batches when a file is too large to load entirely into memory. The default limits allow 64 MiB per compressed or decompressed block and 256 MiB of Arrow buffers per batch; these are not limits on total process memory. Keeping generated records can also retain their input blocks. Adjust limits only after considering the size and trustworthiness of your input.

Malformed or unsupported input raises ValueError. With an iterator, earlier batches may already have been processed before a later error occurs. The limits reference covers configurable bounds and restrictions.

Choosing a Reader​

Tables are a natural fit for column-oriented analysis, while generated records provide typed access for event-by-event processing. The workload determines which is more efficient: representative files and the application's actual projection and decoding requirements provide a useful basis for comparison.

Next Steps​