Skip to main content
Markdown

Protocol Buffers Guide

Generate native Codon message classes from .proto schemas to parse, validate, and serialize Protocol Buffers. Generated classes expose typed fields, nested messages, and mutable collections, with wire-format handling supplied by the Data SDK.

Choose the API according to when the schema becomes available:

APIChoose it when
Generated messagesSchemas are known before compilation and you want typed field access
protobuf.SchemaSchemas are discovered at runtime

Both APIs produce Protobuf wire bytes that can be exchanged with other Protobuf implementations. They do not share message objects with each other. The native application does not need a Python interpreter or the protoc code generator.

Before you begin: install the all-SDK distribution. This tutorial includes the schema and message data, so no external service or dataset is required.

Generate Native Bindings​

Use the tools in your installed data SDK through the Exaloop CLI. Save this schema as common.proto in your working directory:

common.proto
syntax = "proto3";
package example;

message Child {
  string name = 1;
  repeated Child children = 2;
}

Generate its Codon module:

Terminal
mkdir -p build/protobuf-example
exaloop protoc -I. \
  --codon_out=build/protobuf-example common.proto

The distribution registers the SDK automatically. The CLI locates both tools and the bundled standard schemas; use --with NAME before protoc to select another registered SDK. For a custom toolchain, follow individual SDK installation.

This creates build/protobuf-example/common_pb.codon. In the same directory, create main.codon and use its Child class:

main.codon
from common_pb import Child

message = Child()
message.name = "Ada"
message.children.add().name = "Grace"
wire = message.to_bytes()
restored = Child.from_bytes(wire)
assert restored.name == "Ada"
assert restored.children[0].name == "Grace"
Terminal
exaloop --with data run -release -numerics=py build/protobuf-example/main.codon

The example builds a message, adds a child, and round-trips both through the wire format. to_bytes() is the serialization step; from_bytes() constructs and validates a new message from those bytes.

Working with Your Own Schemas​

Each path/name.proto produces path/name_pb.codon. Include imported schemas in the generation command and preserve their directory structure so Codon can resolve the generated imports. The generated package root needs to be importable from your program, just as in the example above.

Module paths follow proto filenames, while class names follow message declarations. A nested declaration becomes a name such as Parent_Child. Recursive messages and cross-file references are supported.

Generated modules import exaloop.data.protobuf_wire. Compile with the matching Data SDK and include its native runtime dependencies in the deployment.

Mutation and Presence​

Generated fields behave like normal attributes, but nested messages and collections stay connected to their parent. In the first example, message.children.add() creates an attached child: assigning its name changes what the parent serializes.

For a singular child, reading the field alone does not mark it present. The first mutation, or an explicit SetInParent(), makes it and its ancestors present. This lets you inspect an empty message without accidentally changing its wire representation. has("field") reports presence for fields whose schema supports it; selecting a oneof member clears the previous selection.

Use add() when building a new repeated child, and append() or extend() when copying existing messages into a collection. Maps have one important Protobuf convention: indexing a missing key creates a default entry. Use get() or a membership test when you only want to look.

If a field is cleared, replaced, or overwritten by a successful parse, references to its old children remain usable but become detached from the parent. Their later mutations do not change the replacement. The generated API covers these behaviors and the available collection operations.

Compatibility and Wire Behavior​

Generated messages exchange ordinary Protobuf wire data with other runtimes. Unknown fields survive parsing and serialization, allowing older applications to forward messages from a newer schema. A failed parse leaves an existing message unchanged.

Encoding the same state repeatedly is deterministic, but the bytes need not be identical to another runtime's encoding. Compare decoded values when checking interoperability. The wire reference covers duplicate fields, enums, and parsing rules.

Current Limits​

This integration generates binary messages, not RPC or gRPC clients. It supports proto2 and proto3 schemas; Editions, extensions, and groups are not supported. See the compatibility reference for size limits and other API differences.

Performance​

Measure the full message-processing path, including parsing, field access, mutation, and serialization where applicable. The profiler helps locate costs in a representative application. Compare unprofiled runs with the same schemas and inputs when evaluating an implementation change.

Next Steps​