RapidProto - fast, header-only Protobuf decoders for C++

~7× faster than protoc + Arena when materializing a full message tree, and faster than protozero when streaming fields - with wire validation that never compiles out (benchmarks).

RapidProto compiles a .proto schema into header-only C++ decoders. One CLI, rapidprotoc, turns your schema into headers you #include. Nothing to link. A single schema gives you two decode models, and you pick whichever fits the job:

Arena. decode() materializes the whole message into a read-only object tree in a bump arena, which you navigate with accessors (person->name()) in any order, as many times as you like.
Streaming. decode() walks the wire once and hands each field’s typed value to a callback you supply. Nothing is materialized, and there’s zero allocation.

A --dump flag adds a third, optional emitter: a debug dumper that prints a decoded arena tree as human-readable, JSON-like text - an inspection aid for logging and debugging, not a spec-compliant JSON codec (see the debug dumper).

Both decode models are decode-only: no serialization, no JSON codec. Both fully validate untrusted wire input (truncation, length overruns, group nesting) and never crash on malformed bytes, and both trust the schema - they assume protoc already accepted it, so field values aren’t range-checked. They cover proto2, proto3, and the newer editions schema format (2023/2024), including groups, maps, and oneofs.

You can read the same schema with either model, and even use both in one translation unit (see using both models).

See architecture.md for the internals and design rationale: the layout planner, the compile-time dispatch, the arena, the coexistence design, and the benchmark methodology.

Why RapidProto

protoc’s C++ runtime is a linked library that builds a full mutable message object per decode; zero-copy pull parsers like protozero drop that allocation but leave you to hand-write the read loop. RapidProto generates the decoder - specialized to your schema at compile time, header-only - and gives you both shapes.

Faster decode. On a realistic mixed payload the arena decoder materializes a full object tree ~7× faster than protoc + google::protobuf::Arena, and the streaming decoder - materializing nothing - is faster still, beating protozero, the zero-copy yardstick, on the realistic payload and most microbenchmark shapes. See benchmarks.
Less memory. The arena tree holds ~half of protoc’s (both payload bytes and total allocation): strings, bytes, and raw payloads are borrowed as views into the input rather than copied, so the arena carries only a read-only bump-allocated tree with no per-field object overhead.
Compile-time typed. Dispatch is entirely compile-time - no std::function, no virtual calls, and a wrong value type or a renamed field is a compile error rather than a silent bug.

RapidProto is decode-only (no serialization, no JSON codec, no reflection) - the one exception being an opt-in debug dumper that emits JSON-like inspection text - the arena tree is read-only, it decodes enums as open even when the schema declares them closed, and it does not validate string UTF-8 (it accepts string bytes protoc would reject). If you also need to produce or mutate messages, keep protoc for that side and use RapidProto for the hot decode path.

Quick start

Requirements: C++17 and a recent GCC or Clang (AppleClang included - Linux and macOS are both CI-covered; MSVC is not supported).

The rapidproto_generate() helper wires generation into a CMake build in a few lines; this section drives the tool by hand so each step is visible. Grab a prebuilt rapidprotoc from the releases page (prebuilt Linux and macOS tarballs, license files included; the macOS binary is unsigned - if Gatekeeper blocks it after extracting, clear the quarantine flag: xattr -d com.apple.quarantine rapidprotoc) - or build it once:

cmake --preset release                               # system compiler, optimized
cmake --build --preset release --target rapidprotoc
# binary: build/release/rapidprotoc

Given person.proto:

syntax = "proto3";
package example;

message Person {
  string name = 1;
  uint32 id = 2;
  repeated string email = 3;   // repeated: navigable array
  Address address = 4;         // sub-message
}

message Address {
  string city = 1;
  string country = 2;
}

1. Generate the arena decoder (the default model) and a self-contained copy of the runtime, into out/:

./build/release/rapidprotoc -I. --out-dir=out person.proto   # add -v to log each written file
# out/person.rp.hpp + out/person.rp.common.hpp + out/rapidproto/{runtime,arena_runtime}.hpp

2. Decode. You supply the serialized message bytes (from a file, socket, database, …) as a rapidproto::ByteView (an alias for std::string_view, so a non-owning view; for a std::uint8_t buffer, rapidproto::byte_view(ptr, size) builds one without a manual cast). Create an Arena, call decode(), then navigate the returned tree:

#include "person.rp.hpp"

namespace ex = rp::arena::example;   // generated types live under rp::arena; alias it once

std::string buf = /* the serialized Person bytes */;

rapidproto::Arena arena;
rapidproto::ArenaDecodeError err;
const ex::Person* p = ex::Person::decode(rapidproto::ByteView(buf), arena, &err);
if (p == nullptr) { /* malformed input: see err.code / err.wire / err.offset */ }

std::uint32_t id = p->id();                        // scalar, by value
std::string_view name = p->name();                 // string, a view into the input buffer
if (const ex::Address* a = p->address())           // sub-message: a pointer (nullptr if absent)
    std::string_view city = a->city();

Test bytes: encode some with protoc - protoc --encode=example.Person -I. person.proto < values.txt > person.bin

3. Compile with only the output directory on the include path:

g++ -std=c++17 -Iout my_consumer.cpp -o my_consumer

Repeated, map and oneof fields, and optional, read differently - see the arena model before writing much against them.

To stream instead, pass --stream (or --arena --stream for both) and use the callback API.

Choosing a model

	Arena	Streaming
What you get	a materialized object tree you read by accessor	a callback fired per field, in wire order
Allocation	one bump arena (you own it)	none
Random access / re-reading	yes: any field, any order, repeatedly	no: a single forward pass
Memory	the whole decoded tree	only what your callbacks keep
`#include`	`<stem>.rp.hpp`	`<stem>.rp.stream.hpp`
Best for	needing the message as a navigable object; a faster/lighter `protoc`+`Arena`	extracting a few fields, stream-processing, lowest overhead

You can use both models for one schema in one translation unit; see using both models.

Documentation

The user manual lives in docs/ (also published at https://veaac.github.io/rapidproto/), one page per topic:

Page	What’s in it
docs/arena.md	The arena decoder: accessors, the `Arena`, `decode_owned`, error handling
docs/streaming.md	The streaming decoder: field tags, the three consumption patterns, aborting
docs/dumper.md	The `--dump` debug dumper: JSON-like inspection text, `DumpOptions`
docs/semantics.md	The shared rules: lifetimes, validation & trust, presence/defaults, open enums, duplicate fields, thread-safety
docs/using-both-models.md	Both models in one TU, the mid-decode hybrid, coexisting with protoc
docs/profiles.md	Decode profiles (`drop` / `raw`) and unknown-field detection (arena)
docs/integration.md	The `rapidprotoc` CLI reference and the CMake helper (incl. cross-compiling)
docs/osm-pbf.md	Real-world walkthrough: OpenStreetMap’s planet format, decoded with both models
docs/optimizations.md	How the decoders got fast: seven optimizations, re-enabled one at a time and measured
docs/benchmarks.md	The numbers, how they’re measured, and how to reproduce them
CHANGELOG.md	Notable user-visible changes per release (SemVer-0: the MINOR version is the breaking axis)
architecture.md	Internals and design rationale, for contributors

A runnable end-to-end example (one schema, both models in one TU, a decode profile) is in examples/consumer.

Compatibility & stability

Versioning is SemVer-0 until 1.0: the MINOR version is the breaking axis - expect breaking changes between 0.x and 0.(x+1), never within a patch, each listed in the CHANGELOG. At 1.0 the promise flips, and the stable surface is everything a consumer binds to: the generated API (names, accessor shapes, callback signatures), the decode-profile file format, the CLI flags, the rapidproto_generate() contract, and the runtime headers’ shape. Only a major release may break or remove anything on that surface. A minor may deprecate (announced in the CHANGELOG under a Deprecated heading, the spelling still working) as advance notice of what the next major removes.

Supported platforms are what CI covers: Linux and macOS, with GCC, Clang and AppleClang. MSVC is not supported - not tested, no workarounds maintained - until real demand shows up.

Contributing

See CONTRIBUTING.md for building, the ./check.sh quality gate, and how the golden tests work. The design and the invariants a change must preserve are in architecture.md.

Security

See SECURITY.md for the threat model and how to report a vulnerability.

License

RapidProto is licensed under the Apache License 2.0; see LICENSE, with attributions in NOTICE and THIRD_PARTY_NOTICES.md.

The vendored runtimes (rapidproto/runtime.hpp, rapidproto/arena_runtime.hpp, and rapidproto/dump_runtime.hpp) carry the same Apache-2.0 license, so the headers rapidprotoc drops into your out-dir are usable under those terms. The decoder code generated from your schema is your own work product, and RapidProto claims no rights over it. The embedded Protocol Buffers well-known-type definitions are Copyright 2008 Google Inc., licensed 3-Clause BSD. Catch2 and protozero are development-time dependencies and are not distributed.