Native and Wire Interfaces
Use these interfaces to exchange Arrow buffers with native libraries or implement schema-independent Protobuf wire tools. They expose layout and ownership contracts that the Data and Protobuf APIs usually manage for you.
Arrow C Data transfers structured column buffers across an ABI boundary. The Protobuf wire helpers operate on encoded fields without a schema. Choose the higher-level APIs unless your integration requires one of these boundaries.
Arrow C Data
ArrowArray and ArrowSchema use the standard
Arrow C Data Interface
layout. The Codon tuple declarations mirror that ABI exactly.
An export keeps the array's storage alive independently of the original handle. Importing that export transfers ownership to the new array:
import exaloop.data as data
with data.array([1, 2, 3]) as original:
exported = original.export_c()
with data.Array.from_c(exported) as imported:
assert imported.to_list(int) == [1, 2, 3]
assert exported.released
The export retains storage after original closes. A successful import transfers
that ownership to the new array and consumes the export's release callbacks,
so exported.released becomes true. Each export can be imported only once.
When exchanging data with an external producer, it must supply valid buffers, offsets, child and dictionary structures, and release callbacks. An export destination must be empty, and borrowed buffers must stay alive until the consumer releases them. This ABI assumes a trusted native producer; it cannot make arbitrary addresses safe to dereference.
Application code exchanges Arrow data through export_c() and Array.from_c().
The standard C Data Interface specifies the layouts and release-callback protocol
used by external producers and consumers. The Data API reference
describes the corresponding Codon handles and their ownership rules.
Protobuf Wire Helpers
The exaloop.data.protobuf_wire module supplies the readers and writers used by generated
codecs. It can also support specialized wire-format tools that need bounded
reads and writes. Unlike a generated message, these helpers have no schema:
the caller chooses legal field tags and is responsible for field types and
presence rules.
| Helper | Contract |
|---|---|
Reader(data: bytes) | Bounded input, maximum 2 GiB minus one |
reader.position, reader.done() | Current byte position and exact-end check |
reader.varint() -> u64, reader.tag() -> int | Checked wire integer/tag reads |
reader.fixed(width) -> u64 | Width must be 4 or 8 |
reader.float32(), reader.float64() | Bit-preserving floating reads |
reader.payload() -> bytes | Read a length-delimited payload with bounds checks |
reader.fixed_values(values: list[T], T) | Append all remaining packed 32/64-bit numeric values |
reader.skip(tag, depth) | Official C++ protobuf unknown-field skipper with recursion bounds |
Writer(capacity: int = 64) | GC-backed growing byte buffer |
writer.varint(value: u64), writer.tag(number, wire) | Write integer/tag; callers supply legal schema tags |
writer.fixed(value: u64, width), float32(value), float64(value) | Little-endian fixed/floating encoding |
writer.fixed_values(values: list[T], T), raw(data: bytes), payload(data: bytes) | Packed, raw, or length-prefixed writes |
writer.finish() -> bytes | Transfer output; reset writer to an empty state |
zigzag_encode(int), zigzag_decode(u64) | Signed zigzag conversion |
varint_size(u64), payload_size(int) | Encoded size calculations |
decode_utf8(data: bytes) -> str | Strict UTF-8 decoding; encode strings with value.encode("utf-8") |
check_depth(depth) | Reject generated nesting beyond 100 |
State, numeric conversion policies, and underscored load/store helpers support
the generator rather than application code. They may change with the generated
layout, so integrations should use the documented wire operations above.
For runtime schema discovery and reflection, the application entry point is
exaloop.data.protobuf.Schema.