-   [](https://platform.exaloop.io/)
-   Data and messages
-   Protocol Buffers Guide

# Protocol Buffers Guide

Generate native Codon message classes from `.proto` schemas to parse, validate, and serialize Protocol Buffers. Generated classes expose typed fields, nested messages, and mutable collections, with wire-format handling supplied by the Data SDK.

Choose the API according to when the schema becomes available:

| API | Choose it when |
| --- | --- |
| Generated messages | Schemas are known before compilation and you want typed field access |
| `protobuf.Schema` | Schemas are discovered at runtime |

Both APIs produce Protobuf wire bytes that can be exchanged with other Protobuf implementations. They do not share message objects with each other. The native application does not need a Python interpreter or the `protoc` code generator.

**Before you begin:** install the [all-SDK distribution](https://platform.exaloop.io/start/distribution/). This tutorial includes the schema and message data, so no external service or dataset is required.

## Generate Native Bindings

Use the tools in your installed [data SDK](https://platform.exaloop.io/start/install/#data-plugin) through the [Exaloop CLI](https://platform.exaloop.io/start/cli/#sdk-tools). Save this schema as `common.proto` in your working directory:

```protobuf title="common.proto"
syntax = "proto3";
package example;

message Child {
  string name = 1;
  repeated Child children = 2;
}
```

Generate its Codon module:

```sh title="Terminal"
mkdir -p build/protobuf-example
exaloop protoc -I. \
  --codon_out=build/protobuf-example common.proto
```

The distribution registers the SDK automatically. The CLI locates both tools and the bundled standard schemas; use `--with NAME` before `protoc` to select another registered SDK. For a custom toolchain, follow [individual SDK installation](https://platform.exaloop.io/start/install/).

This creates `build/protobuf-example/common_pb.codon`. In the same directory, create `main.codon` and use its `Child` class:

```python title="main.codon"
from common_pb import Child

message = Child()
message.name = "Ada"
message.children.add().name = "Grace"
wire = message.to_bytes()
restored = Child.from_bytes(wire)
assert restored.name == "Ada"
assert restored.children[0].name == "Grace"
```

```sh title="Terminal"
exaloop --with data run -release -numerics=py build/protobuf-example/main.codon
```

The example builds a message, adds a child, and round-trips both through the wire format. `to_bytes()` is the serialization step; `from_bytes()` constructs and validates a new message from those bytes.

### Working with Your Own Schemas

Each `path/name.proto` produces `path/name_pb.codon`. Include imported schemas in the generation command and preserve their directory structure so Codon can resolve the generated imports. The generated package root needs to be importable from your program, just as in the example above.

Module paths follow proto filenames, while class names follow message declarations. A nested declaration becomes a name such as `Parent_Child`. Recursive messages and cross-file references are supported.

Generated modules import `exaloop.data.protobuf_wire`. Compile with the matching Data SDK and include its native runtime dependencies in the deployment.

## Mutation and Presence

Generated fields behave like normal attributes, but nested messages and collections stay connected to their parent. In the first example, `message.children.add()` creates an attached child: assigning its `name` changes what the parent serializes.

For a singular child, reading the field alone does not mark it present. The first mutation, or an explicit `SetInParent()`, makes it and its ancestors present. This lets you inspect an empty message without accidentally changing its wire representation. `has("field")` reports presence for fields whose schema supports it; selecting a oneof member clears the previous selection.

Use `add()` when building a new repeated child, and `append()` or `extend()` when copying existing messages into a collection. Maps have one important Protobuf convention: indexing a missing key creates a default entry. Use `get()` or a membership test when you only want to look.

If a field is cleared, replaced, or overwritten by a successful parse, references to its old children remain usable but become detached from the parent. Their later mutations do not change the replacement. The [generated API](https://platform.exaloop.io/data/protobuf-api/) covers these behaviors and the available collection operations.

## Compatibility and Wire Behavior

Generated messages exchange ordinary Protobuf wire data with other runtimes. Unknown fields survive parsing and serialization, allowing older applications to forward messages from a newer schema. A failed parse leaves an existing message unchanged.

Encoding the same state repeatedly is deterministic, but the bytes need not be identical to another runtime's encoding. Compare decoded values when checking interoperability. The [wire reference](https://platform.exaloop.io/data/protobuf-api/#wire-compatibility) covers duplicate fields, enums, and parsing rules.

### Current Limits

This integration generates binary messages, not RPC or gRPC clients. It supports proto2 and proto3 schemas; Editions, extensions, and groups are not supported. See the [compatibility reference](https://platform.exaloop.io/data/protobuf-api/#wire-compatibility) for size limits and other API differences.

## Performance

Measure the full message-processing path, including parsing, field access, mutation, and serialization where applicable. The [profiler](https://platform.exaloop.io/start/profiling/) helps locate costs in a representative application. Compare unprofiled runs with the same schemas and inputs when evaluating an implementation change.

## Next Steps

-   Use the [generated API reference](https://platform.exaloop.io/data/protobuf-api/) for maps, repeated fields, and presence checks.
-   [Load schemas at runtime](https://platform.exaloop.io/data/protobuf-dynamic/) when they are not known before compilation.
-   [Send serialized messages to Kafka](https://platform.exaloop.io/streaming/kafka/) from a native producer.

Source: https://platform.exaloop.io/data/protobuf/