Protocol Buffers (protobuf) — User Guide

1. Quick Start

Protocol Buffers (protobuf) is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You define how your data is structured once in .proto files, then use the protoc compiler to generate source code in your target language — C++, Java, Python, C#, Go, Ruby, PHP, Kotlin, Objective-C, Rust, or Dart. The generated code provides typed accessors for reading and writing your data efficiently in a compact binary format.

Key features:

• Compact binary serialization — 3–10x smaller and 20–100x faster than XML
• Strongly typed schema with forward and backward compatibility
• Code generation for 12+ programming languages from a single .proto definition
• Built-in JSON mapping for interoperability with web services
• Well-known types for common patterns (timestamps, durations, field masks, etc.)
• Editions system for evolving language features without breaking changes
• Conformance testing framework to verify cross-language compatibility
• Lite runtime for resource-constrained environments (Android, embedded)

Prerequisites: A working compiler toolchain for your target language (e.g., JDK 8+ for Java, Python 3.x for Python, .NET SDK for C#).

Install protoc and try it:

# Download the latest release for your platform
# from https://github.com/protocolbuffers/protobuf/releases
# Extract and add to PATH

protoc --version

libprotoc 4.36.0

Minimal example — define a message, generate code, use it:

Create person.proto:

syntax = "proto3";
package example;

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}

Generate Python code:

protoc --python_out=. person.proto

Use the generated code:

import person_pb2

person = person_pb2.Person()
person.name = "Alice"
person.id = 42
person.email = "[email removed]"

# Serialize to binary
data = person.SerializeToString()

# Deserialize
person2 = person_pb2.Person()
person2.ParseFromString(data)
print(person2.name)  # Alice

Alice

Tip

The fastest way to get started is to download a prebuilt protoc binary from the GitHub Releases page and install the runtime library for your language via its package manager.

What to try next:

• See the Installation section for all supported install methods
• See Core Usage for the complete .proto language guide
• See Common Tasks for real-world recipes (JSON conversion, well-known types, maps)

2. Installation

System Requirements

Installing protoc (the Protocol Buffer Compiler)

Pre-built Binary (recommended)

# Download from GitHub releases
# https://github.com/protocolbuffers/protobuf/releases
# Choose protoc-<version>-<platform>.zip

# Linux x86_64 example:
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v36.0/protoc-36.0-linux-x86_64.zip
unzip protoc-36.0-linux-x86_64.zip -d $HOME/.local
export PATH="$HOME/.local/bin:$PATH"

protoc --version

libprotoc 4.36.0

Homebrew (macOS/Linux)

brew install protobuf
protoc --version

apt (Debian/Ubuntu)

sudo apt install -y protobuf-compiler
protoc --version

Build from Source (Bazel)

git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
bazel build //:protoc
# Binary at bazel-bin/protoc

Build from Source (CMake)

git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target protoc
# Binary at build/protoc

Important

When using the Rust crate, the protoc binary version must exactly match the protobuf crate version. If you are using Rust protobuf x.y.z, you need protoc y.z. Download the matching release from the GitHub Releases page.

Installing Language-Specific Runtime Libraries

Java Maven dependency:

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>4.36.0</version>
</dependency>

Java BOM (Bill of Materials):

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-bom</artifactId>
      <version>4.36.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Warning

Distribution packages (e.g., apt install protobuf-compiler) often ship older versions of protoc. For production use, prefer the pre-built binaries from GitHub Releases to ensure version compatibility between the compiler and runtime libraries.

Post-Install Verification

protoc --version

libprotoc 4.36.0

Verify language runtimes:

# Python
python -c "import google.protobuf; print(google.protobuf.__version__)"

4.36.0

# Ruby
ruby -e "require 'google/protobuf'; puts Google::Protobuf::VERSION"

Install Troubleshooting

protoc: command not found — the binary is not in your PATH. Add the directory containing protoc to your PATH environment variable.

protoc: error while loading shared libraries — on Linux, you may need to run ldconfig after installing or set LD_LIBRARY_PATH to the library directory.

Version mismatch between protoc and runtime — always ensure the protoc compiler version matches the runtime library version. Mixing versions can cause subtle serialization bugs or compilation errors.

pip install protobuf fails on Apple Silicon — try pip install --no-binary protobuf protobuf to build from source, or ensure you have the latest pip and setuptools.

CMake build fails with abseil-cpp errors — protobuf depends on abseil-cpp. Ensure you initialized submodules: git submodule update --init --recursive. See GitHub issue #22209.

npm install google-protobuf — the JavaScript implementation lives in the separate protobuf-javascript repository.

Go module errors — the Go runtime lives in the separate protobuf-go repository at google.golang.org/protobuf.

Dart protobuf — the Dart implementation lives in the separate dart-lang/protobuf repository.

Windows: protoc.exe is blocked by antivirus — add an exception for the protoc binary or download from the official GitHub Releases page.

gcc compilation failures (issue #26385) — building protobuf from source with certain GCC versions may fail. Use a supported compiler version or build with Clang.

Ruby JRuby vs CRuby — on JRuby, the Ruby gem uses FFI to access the protobuf library. On CRuby, it uses a compiled C extension backed by upb. Both are installed via gem install google-protobuf.

3. Configuration

Proto File Syntax

Protocol Buffers uses .proto files to define message schemas. The current recommended syntax is proto3, with proto2 supported for legacy schemas and the new Editions system for evolving features.

Minimal proto file:

syntax = "proto3";
package mypackage;

message MyMessage {
  string name = 1;
  int32 count = 2;
  bool active = 3;
}

Scalar Types

Language-Specific Options

Common file-level options you can set in .proto files:

// Java
option java_package = "com.example.myproject";
option java_outer_classname = "MyProtos";
option java_multiple_files = true;

// C#
option csharp_namespace = "MyCompany.MyProject";

// Go
option go_package = "github.com/myorg/myrepo/mypb";

// Objective-C
option objc_class_prefix = "MYP";

// Ruby
option ruby_package = "MyCompany::MyProject";

// PHP
option php_namespace = "MyCompany\\MyProject";
option php_metadata_namespace = "MyCompany\\MyProject\\Metadata";

Python Backend Selection

The Python protobuf library supports multiple backends. The default backend (upb) is the fastest:

Note

The cpp backend is deprecated. The upb backend (default since protobuf v4) offers equivalent or better performance. Setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to any value other than python, cpp, or upb raises a ValueError.

See the Admin Guide for the full configuration reference including all protoc flags, edition features, and build system options.

4. Core Usage

Defining Messages

Messages are the fundamental building blocks of protobuf schemas. Each field has a type, a name, and a unique field number.

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }

  repeated PhoneNumber phones = 4;
  google.protobuf.Timestamp last_updated = 5;
}

message AddressBook {
  repeated Person people = 1;
}

Tip

Field numbers 1–15 use a single byte in the wire format, so assign them to frequently populated fields for optimal serialization size.

Field Rules

Enumerations

enum Status {
  STATUS_UNSPECIFIED = 0;  // First value MUST be zero
  ACTIVE = 1;
  INACTIVE = 2;
  DELETED = 3;
}

Important

In proto3, the first enum value must be zero and serves as the default. Name it with an _UNSPECIFIED suffix by convention.

Maps

message Project {
  map<string, int32> scores = 1;
  map<int32, Person> people_by_id = 2;
}

Map key types can be any integer or string type. Map value types can be any type except another map.

Oneof Fields

message Profile {
  string name = 1;
  oneof avatar {
    string image_url = 2;
    bytes image_data = 3;
  }
}

Setting any field within a oneof clears all other fields in the group.

Importing Other Proto Files

import "google/protobuf/timestamp.proto";
import "myother/messages.proto";

Generating Code

Use protoc to compile .proto files into language-specific source code:

# Python
protoc --python_out=./gen person.proto

# Java
protoc --java_out=./gen person.proto

# C++
protoc --cpp_out=./gen person.proto

# C#
protoc --csharp_out=./gen person.proto

# Ruby
protoc --ruby_out=./gen person.proto

# PHP
protoc --php_out=./gen person.proto

# Kotlin (requires --java_out too on JVM)
protoc --java_out=./gen --kotlin_out=./gen person.proto

# Objective-C
protoc --objc_out=./gen person.proto

# Multiple outputs at once
protoc --java_out=./java_gen --python_out=./py_gen person.proto

Specifying import paths:

protoc -I=./protos -I=./third_party --python_out=./gen ./protos/person.proto

Note

The -I (or --proto_path) flag tells protoc where to search for imported .proto files. You can specify it multiple times for multiple search paths.

Using Generated Code

Python

import person_pb2

# Create and populate
person = person_pb2.Person()
person.name = "Bob"
person.id = 1234
person.email = "[email removed]"

phone = person.phones.add()
phone.number = "555-4321"
phone.type = person_pb2.Person.HOME

# Serialize to binary
data = person.SerializeToString()

# Deserialize from binary
person2 = person_pb2.Person()
person2.ParseFromString(data)
print(person2.name)

Bob

Warning

In Python, assigning to a repeated or composite field directly (e.g., person.phones = [...]) raises an AttributeError. Use .add(), .append(), .extend(), or .CopyFrom() instead.

Java

import com.example.tutorial.protos.Person;
import com.example.tutorial.protos.AddressBook;
import com.google.protobuf.util.Timestamps;

// Create and populate
Person person = Person.newBuilder()
    .setName("Bob")
    .setId(1234)
    .setEmail("[email removed]")
    .addPhones(Person.PhoneNumber.newBuilder()
        .setNumber("555-4321")
        .setType(Person.PhoneType.HOME))
    .setLastUpdated(Timestamps.now())
    .build();

// Serialize to bytes
byte[] data = person.toByteArray();

// Deserialize from bytes
Person person2 = Person.parseFrom(data);

// Write to file
try (FileOutputStream output = new FileOutputStream("person.bin")) {
    person.writeTo(output);
}

// Read from file
Person person3 = Person.parseFrom(new FileInputStream("person.bin"));

C++

#include "person.pb.h"
#include <fstream>

tutorial::Person person;
person.set_name("Bob");
person.set_id(1234);
person.set_email("[email removed]");

auto* phone = person.add_phones();
phone->set_number("555-4321");
phone->set_type(tutorial::Person::HOME);

// Serialize
std::string data;
person.SerializeToString(&data);

// Deserialize
tutorial::Person person2;
person2.ParseFromString(data);

// Write to file
std::ofstream output("person.bin", std::ios::binary);
person.SerializeToOstream(&output);

C#

using Google.Protobuf;
using Tutorial;

var person = new Person
{
    Name = "Bob",
    Id = 1234,
    Email = "[email removed]",
    Phones = { new Person.Types.PhoneNumber
    {
        Number = "555-4321",
        Type = Person.Types.PhoneType.Home
    }}
};

// Serialize
byte[] data = person.ToByteArray();

// Deserialize
Person person2 = Person.Parser.ParseFrom(data);

Ruby

require 'google/protobuf'
require 'addressbook_pb'

person = Tutorial::Person.new(
  name: "Bob",
  id: 1234,
  email: "[email removed]"
)
person.phones << Tutorial::Person::PhoneNumber.new(
  number: "555-4321",
  type: :HOME
)

# Serialize
data = Tutorial::Person.encode(person)

# Deserialize
person2 = Tutorial::Person.decode(data)

PHP

use Google\Protobuf\Internal\Message;
use Tutorial\Person;
use Tutorial\Person\PhoneNumber;
use Tutorial\Person\PhoneType;

$person = new Person();
$person->setName("Bob");
$person->setId(1234);
$person->setEmail("[email removed]");

$phone = new PhoneNumber();
$phone->setNumber("555-4321");
$phone->setType(PhoneType::HOME);
$person->setPhones([$phone]);

// Serialize
$data = $person->serializeToString();

// Deserialize
$person2 = new Person();
$person2->mergeFromString($data);

Rust

use protobuf::proto;

// Using the proto! macro
let person = proto!(Person {
    name: "Bob",
    id: 1234,
    email: "[email removed]"
});

// Serialize
let data = person.serialize().unwrap();

// Deserialize
let person2 = Person::parse(&data).unwrap();
assert_eq!(person2.name(), "Bob");

Note

The Rust protobuf crate (V4) is backed by either the upb C library or the C++ protobuf implementation. It provides a safe Rust API with View, Mut, and owned message types for zero-copy access patterns.

Go

import (
    "google.golang.org/protobuf/proto"
    tutorialpb "github.com/protocolbuffers/protobuf/examples/go/tutorialpb"
)

person := &tutorialpb.Person{
    Name:  "Bob",
    Id:    1234,
    Email: "[email removed]",
}

// Serialize
data, err := proto.Marshal(person)

// Deserialize
person2 := &tutorialpb.Person{}
err = proto.Unmarshal(data, person2)

5. Common Tasks

Task 1: Converting Between Protobuf and JSON

Goal: Serialize protobuf messages to JSON for REST APIs or debugging.

Python:

from google.protobuf.json_format import MessageToJson, Parse

# To JSON
json_string = MessageToJson(person)
print(json_string)

# From JSON
person2 = person_pb2.Person()
Parse(json_string, person2)

{
  "name": "Bob",
  "id": 1234,
  "email": "[email removed]"
}

Java:

import com.google.protobuf.util.JsonFormat;

// To JSON
String json = JsonFormat.printer().print(person);

// From JSON
Person.Builder builder = Person.newBuilder();
JsonFormat.parser().merge(json, builder);
Person person2 = builder.build();

PHP:

// To JSON
$json = $person->serializeToJsonString();

// From JSON
$person2 = new Person();
$person2->mergeFromJsonString($json);

Tip

JSON output uses camelCase field names by default (e.g., last_updated becomes lastUpdated). Both naming conventions are accepted when parsing.

Task 2: Using Well-Known Types

Protobuf includes a set of well-known types for common patterns. Import them from google/protobuf/:

Using Timestamp (Python):

from google.protobuf.timestamp_pb2 import Timestamp
from datetime import datetime

ts = Timestamp()
ts.GetCurrentTime()

# From datetime
ts.FromDatetime(datetime(2026, 1, 15, 10, 30, 0))

# To datetime
dt = ts.ToDatetime()

Warning

Timestamp seconds must be in range [-62135596800, 253402300799] and nanos in [0, 999999999]. Values outside these ranges raise a ValueError.

Using Duration (Python):

from google.protobuf.duration_pb2 import Duration
from datetime import timedelta

dur = Duration()
dur.FromTimedelta(timedelta(hours=2, minutes=30))
td = dur.ToTimedelta()

Using Any (Java):

import com.google.protobuf.Any;

Any any = Any.pack(person);

if (any.is(Person.class)) {
    Person unpacked = any.unpack(Person.class);
}

Task 3: Working with Maps

message GameState {
  map<string, int32> player_scores = 1;
  map<int32, string> id_to_name = 2;
}

Python:

game = game_pb2.GameState()
game.player_scores["alice"] = 100
game.player_scores["bob"] = 85

for name, score in game.player_scores.items():
    print(f"{name}: {score}")

Note

Map fields cannot be repeated. Map iteration order is not guaranteed to be consistent across languages or serialization round-trips.

Task 4: Using Oneof for Variant Types

message PaymentMethod {
  oneof method {
    CreditCard card = 1;
    BankTransfer transfer = 2;
    string voucher_code = 3;
  }
}

Python:

payment = PaymentMethod()
payment.voucher_code = "SAVE20"

# Check which field is set
which = payment.WhichOneof("method")
print(which)  # "voucher_code"

# Setting a different oneof field clears the previous one
payment.card.CopyFrom(some_card)
print(payment.HasField("voucher_code"))  # False

Task 5: Schema Evolution — Adding and Removing Fields

Adding a new field (backward compatible):

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
  string nickname = 4;  // NEW — old code ignores this field
}

Removing a field (use reserved to prevent reuse):

message Person {
  reserved 3;
  reserved "email";
  string name = 1;
  int32 id = 2;
  // email was field 3, now removed
}

Important

Never reuse a field number after removing a field — old serialized data still uses those numbers. Always mark removed field numbers and names as reserved.

Task 6: Writing Proto Files with Imports and Packages

syntax = "proto3";
package mycompany.myproject;

import "google/protobuf/timestamp.proto";
import "common/shared_types.proto";

message Order {
  string id = 1;
  common.Customer customer = 2;
  repeated common.LineItem items = 3;
  google.protobuf.Timestamp created_at = 4;
}

Compile with import paths:

protoc \
  -I=./protos \
  -I=./third_party \
  --java_out=./gen \
  ./protos/mycompany/myproject/order.proto

Task 7: Working with Repeated Fields (Lists)

Python:

person = person_pb2.Person()

# Add items
phone = person.phones.add()
phone.number = "555-1234"
phone.type = person_pb2.Person.MOBILE

# Add multiple
person.phones.extend([
    person_pb2.Person.PhoneNumber(number="555-5678", type=person_pb2.Person.WORK)
])

# Iterate
for phone in person.phones:
    print(phone.number)

# Length
print(len(person.phones))

Java:

Person.Builder builder = Person.newBuilder();
builder.addPhones(Person.PhoneNumber.newBuilder()
    .setNumber("555-1234")
    .setType(Person.PhoneType.MOBILE));

// Get count
int count = builder.getPhonesCount();

// Get by index
Person.PhoneNumber first = builder.getPhones(0);

// Get list
List<Person.PhoneNumber> all = builder.getPhonesList();

Task 8: Checking Field Presence

In proto3, singular scalar fields return their default value when not set. Use optional to track presence explicitly:

message Settings {
  optional int32 timeout = 1;     // Has presence
  int32 retries = 2;              // No presence (default = 0)
}

Python:

settings = settings_pb2.Settings()

# Check if optional field was explicitly set
if settings.HasField("timeout"):
    print(settings.timeout)

# For oneof
which = msg.WhichOneof("my_oneof")

Java:

if (settings.hasTimeout()) {
    int timeout = settings.getTimeout();
}

Task 9: Using Extensions (proto2 only)

syntax = "proto2";

message MyMessage {
  extensions 100 to 199;
}

extend MyMessage {
  optional string my_extension = 100;
}

Task 10: Serializing to and from Files

Python:

# Write
with open("data.bin", "wb") as f:
    f.write(address_book.SerializeToString())

# Read
address_book = addressbook_pb2.AddressBook()
with open("data.bin", "rb") as f:
    address_book.ParseFromString(f.read())

Java:

// Write
try (FileOutputStream out = new FileOutputStream("data.bin")) {
    addressBook.writeTo(out);
}

// Read
AddressBook book = AddressBook.parseFrom(new FileInputStream("data.bin"));

Ruby:

# Write
File.open("data.bin", "wb") { |f| f.write(Tutorial::AddressBook.encode(book)) }

# Read
book = Tutorial::AddressBook.decode(File.read("data.bin", mode: "rb"))

Task 11: Using Kotlin DSL

When targeting the JVM, Kotlin provides a DSL for building messages:

val person = person {
    name = "Alice"
    id = 42
    email = "[email removed]"
    phones += phoneNumber {
        number = "555-1234"
        type = Person.PhoneType.MOBILE
    }
}

Note

The Kotlin DSL requires both --java_out and --kotlin_out flags when running protoc. It sits on top of the Java generated code.

Task 12: Merging Messages

Python:

# MergeFrom: existing fields are overwritten, repeated fields are appended
target = person_pb2.Person()
target.MergeFrom(source)

# CopyFrom: clears target first, then copies
target.CopyFrom(source)

Java:

Person.Builder builder = Person.newBuilder();
builder.mergeFrom(existingPerson);

Task 13: Comparing Messages

Python:

if person1 == person2:
    print("Messages are equal")

Java:

if (person1.equals(person2)) {
    System.out.println("Messages are equal");
}

Task 14: Working with Unknown Fields

When a message is deserialized with fields that don't match the schema, those fields are preserved as unknown fields and re-serialized. This is key to forward compatibility.

Python:

# Access unknown fields (read-only)
from google.protobuf.internal.containers import UnknownFieldSet

unknown = UnknownFieldSet(message)
for field in unknown:
    print(field.field_number, field.wire_type, field.data)

Task 15: Using the Lite Runtime (Android/Embedded)

For resource-constrained environments, use the lite runtime which produces smaller generated code:

Java (Maven):

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-javalite</artifactId>
  <version>4.36.0</version>
</dependency>

The lite runtime omits reflection, descriptors, and text format support but provides full serialization/deserialization.

Task 16: Text Format (Debugging)

Python:

from google.protobuf import text_format

# To text format
text = text_format.MessageToString(person)
print(text)

# From text format
person2 = person_pb2.Person()
text_format.Parse(text, person2)

name: "Bob"
id: 1234
email: "[email removed]"
phones {
  number: "555-4321"
  type: HOME
}

Task 17: Defining Services (for gRPC)

syntax = "proto3";

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
  rpc SayHelloStream (HelloRequest) returns (stream HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}

Generate gRPC stubs alongside protobuf code using language-specific gRPC plugins.

Task 18: Using Editions (replacing proto2/proto3 syntax)

Editions are the modern way to evolve protobuf language features:

edition = "2023";
package mypackage;

message MyMessage {
  string name = 1;
  int32 id = 2;
}

Editions replace the syntax keyword and allow gradual feature migration via feature flags.

Task 19: Building with Bazel

# BUILD file
load("@rules_proto//proto:proto_library.bzl", "proto_library")
load("@protobuf//bazel:java_proto_library.bzl", "java_proto_library")

proto_library(
    name = "person_proto",
    srcs = ["person.proto"],
)

java_proto_library(
    name = "person_java_proto",
    deps = [":person_proto"],
)

Task 20: Default Values

In proto3, fields have language-specific default values when not set:

6. Troubleshooting

Diagnostic Decision Tree

Problem with protobuf?
├── Compilation error in .proto file?
│   ├── Check syntax declaration (proto2/proto3/edition)
│   ├── Check field numbers are unique within message
│   ├── Check enum first value is zero (proto3)
│   └── Check import paths match -I flags
├── Generated code won't compile?
│   ├── Version mismatch: protoc vs runtime library
│   ├── Missing imports in generated code
│   └── Language-specific build config issue
├── Serialization/deserialization failure?
│   ├── Schema mismatch between writer and reader
│   ├── Truncated data (incomplete read)
│   └── Encoding issue (string not valid UTF-8)
├── Performance issue?
│   ├── Python: check backend (upb vs pure python)
│   ├── Large messages: consider streaming
│   └── Repeated parsing: reuse message objects
└── Runtime error?
    ├── Check error message in Troubleshooting below
    └── Check GitHub issues

Error: Truncated message.

Cause: The binary data being parsed is incomplete — the parser reached the end of input before the message was fully decoded.

Solution:

1. Verify the data was fully written (check file size, network transfer)
2. Ensure you're reading in binary mode ("rb" in Python)
3. Check for buffer size issues when reading from streams

Prevention: Always verify the byte count matches between serializer and parser.

Error: Too many bytes when decoding varint.

Cause: A variable-length integer (varint) in the binary data exceeded the maximum 10-byte encoding.

Solution:

1. Check for data corruption
2. Verify the data is actually protobuf-encoded (not JSON or text format)
3. Ensure you're not accidentally reading text-format data as binary

Error: DecodeError: Field number 0 is illegal.

Cause: The parser encountered a field with number 0, which is reserved by the wire format.

Solution:

1. Check for data corruption or incorrect offset when reading
2. Ensure the data is protobuf binary format, not JSON or text

Error: Missing group end tag.

Cause: A group-encoded field (legacy proto2 feature) is missing its terminator.

Solution:

1. Verify data integrity
2. If migrating from proto2, ensure groups are properly closed

Error: Unexpected end-group tag.

Cause: An end-group wire type appeared where the parser didn't expect one.

Solution:

1. Check for schema mismatch between serializer and deserializer
2. Verify data isn't truncated or corrupted

Error: Error parsing message: too many levels of nesting.

Cause: The message exceeds the recursion depth limit (default: 100 levels).

Solution:

1. Flatten deeply nested message structures
2. Increase the recursion limit via CodedInputStream.setRecursionLimit() (Java) if the nesting is intentional

Prevention: Design schemas with reasonable nesting depth — deeply nested structures indicate a design problem.

Error: Message is missing required fields (proto2)

Cause: A proto2 required field was not set before serialization.

Solution:

1. Set all required fields before calling SerializeToString()
2. Use IsInitialized() to check before serializing

Error: Protocol message has no "X" field.

Cause: Attempting to access a field that doesn't exist in the message descriptor.

Solution:

1. Check for typos in field names
2. Verify you're using the correct generated class version
3. Regenerate code if the .proto file was recently updated

Error: Cannot set X to Y: TypeError

Cause: Attempting to assign a value of the wrong type to a field (e.g., string to int32).

Solution:

1. Check the field type in the .proto file
2. Convert the value to the correct type before assignment

Error: Assignment not allowed to repeated field

Cause: In Python, directly assigning to a repeated field (e.g., msg.items = [...]).

Solution: Use mutation methods instead:

msg.items.extend([item1, item2])
# or
msg.items.append(item)
# or
del msg.items[:]
msg.items.extend(new_items)

Error: Assignment not allowed to composite field

Cause: In Python, directly assigning to a sub-message field.

Solution: Use CopyFrom() or MergeFrom():

msg.sub_message.CopyFrom(other_sub_message)

Error: Enum has no value defined for name

Cause: Looking up an enum value by a name that doesn't exist.

Solution:

1. Check for typos in the enum value name
2. Verify the .proto file contains the expected enum values
3. Regenerate code after proto file changes

Error: Value out of range for integer fields

Cause: Assigning a value outside the valid range for the field type.

Solution: Check the range for your field type:

• int32: -2,147,483,648 to 2,147,483,647
• uint32: 0 to 4,294,967,295
• int64: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807

Error: OOM in packed field decoders (GitHub issue #27104)

Cause: Maliciously crafted messages with extremely large packed field counts can cause out-of-memory conditions.

Solution:

1. Set message size limits when parsing untrusted input
2. Use CodedInputStream.setSizeLimit() (Java/C++) to cap message size

Caution

Always set size limits when parsing protobuf messages from untrusted sources. Without limits, a malicious message can consume all available memory.

Error: Null Pointer Dereference in EpsCopyOutputStream::EnableAliasing (GitHub issue #27099)

Cause: Using the array-only constructor with EnableAliasing can trigger a null pointer dereference.

Solution:

1. Avoid the array-only constructor pattern when aliasing is enabled
2. Update to a patched version when available

Error: Allocation-size-too-big crash in GzipOutputStream::Init (GitHub issue #27097)

Cause: A negative buffer_size parameter in GzipOutputStream::Init is not validated, causing an allocation failure.

Solution:

1. Ensure buffer sizes are always positive
2. Update to a patched version when available

Error: PHP GPBDecodeException: Fail to push limit.

Cause: Malformed or truncated protobuf data being parsed by the PHP runtime.

Solution:

1. Verify the data source is producing valid protobuf binary format
2. Check for data truncation during transmission

Error: PHP GPBDecodeException: Illegal field number zero.

Cause: Corrupted protobuf data containing an invalid field number.

Solution:

1. Verify data integrity
2. Ensure the correct binary data is being passed to mergeFromString()

Error: PHP GPBDecodeException: Unexpected wire type.

Cause: The wire type in the serialized data doesn't match the expected type for the field.

Solution:

1. Check for schema version mismatch between producer and consumer
2. Verify data isn't corrupted or mixed with non-protobuf data

Error: PHP GPBDecodeException: Invalid base64 characters

Cause: Base64-encoded protobuf data contains invalid characters when parsed via JSON.

Solution:

1. Ensure proper base64 encoding of bytes fields
2. Use standard base64 encoding (not URL-safe) unless the parser expects it

Error: RuntimeError: The generated code only work with python cpp extension

Cause: Generated Python code was produced for the C++ extension backend but the pure Python runtime is active.

Solution:

1. Regenerate the Python code
2. Or set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb (default)

Error: PickleError: Can't pickle repeated scalar fields

Cause: Attempting to pickle a RepeatedScalarFieldContainer directly.

Solution: Convert to a list first:

import pickle
my_list = list(msg.repeated_field)
pickle.dumps(my_list)

Error: FieldMask validation errors

Cause: Invalid field mask path names — paths must use lowercase with underscores.

Solution:

1. Path names must not contain uppercase letters when printing to JSON
2. Path names must not contain underscores when parsing from JSON (use camelCase)
3. Trailing underscores in path names are not allowed

Error: Duration is not valid: Sign mismatch.

Cause: Duration seconds and nanos have different signs.

Solution: Ensure both seconds and nanos are either both non-negative or both non-positive:

• Valid: seconds=5, nanos=500000000
• Invalid: seconds=5, nanos=-500000000

Warning

Duration seconds must be in range [-315576000000, 315576000000] (approximately ±10,000 years) and nanos in [-999999999, 999999999].

Error: Timestamp JSON value not a string

Cause: Parsing a Timestamp from JSON but the value is not a string.

Solution: Timestamp JSON format requires RFC 3339 strings:

"2026-01-15T10:30:00Z"

Error: Duration must end with letter "s"

Cause: Parsing a Duration from JSON but the value doesn't end with s.

Solution: Duration JSON format requires a string ending in s:

"3600s"

7. FAQ

General

Q: What is the difference between proto2 and proto3?

Proto3 simplifies the language by removing required fields, making all fields optional by default, and dropping default value customization. Proto3 is the recommended syntax for new projects. Proto2 remains supported for existing schemas. The new Editions system is the forward-looking replacement for both.

Q: What are Editions and should I use them?

Editions replace the syntax = "proto2" / syntax = "proto3" declaration with edition = "2023" (or later). They allow gradual, per-feature migration without a hard syntax boundary. For new projects, either proto3 or edition 2023 is recommended.

Q: Which languages does protobuf support?

First-party support (in this repository): C++, Java, Python, C#, Objective-C, Ruby, PHP, Kotlin, Rust. First-party but separate repos: Go (protobuf-go), Dart (dart-lang/protobuf), JavaScript (protobuf-javascript). Many community-maintained implementations exist for other languages.

Q: Is protobuf backward/forward compatible?

Yes, by design. New fields can be added without breaking old readers (they ignore unknown fields). Old fields can be removed if their numbers are reserved. Never reuse field numbers.

Q: What is the maximum message size?

The default limit is 64 MB (configurable). For larger payloads, consider streaming or splitting into multiple messages.

Schema Design

Q: Should I use int32 or sint32?

Use int32 for values that are usually non-negative. Use sint32 for values that are frequently negative — it uses ZigZag encoding which is more efficient for negative numbers. Same applies to int64 vs sint64.

Q: When should I use fixed32/fixed64 instead of int32/int64?

Use fixed32/fixed64 when values are typically large (> 2^28 or 2^56) — they use a constant 4 or 8 bytes, which is more efficient than varint encoding for large values.

Q: Can I use float or double as map keys?

No. Map keys can only be integer types (int32, int64, uint32, uint64, sint32, sint64, fixed32, fixed64, sfixed32, sfixed64), bool, or string.

Q: How do I represent nullable fields in proto3?

Use optional to track field presence, or use wrapper types from google/protobuf/wrappers.proto (e.g., google.protobuf.Int32Value).

Q: What field numbers should I avoid?

Field numbers 19000–19999 are reserved by the protobuf implementation. Field number 0 is invalid. Numbers 1–15 use 1 byte on the wire, 16–2047 use 2 bytes.

Q: Can map fields be repeated?

No. repeated map<K,V> is not valid protobuf syntax. Use a message wrapper if you need a list of maps.

Runtime and Performance

Q: Which Python backend should I use?

Use the default upb backend. It's C-based and provides the best performance. The cpp backend is deprecated and the python (pure Python) backend is significantly slower.

Q: What is the Lite runtime?

The lite runtime (protobuf-javalite for Java) produces smaller generated code by omitting reflection, descriptors, and text format support. Use it for Android apps and other size-sensitive environments.

Q: How do I improve serialization performance?

1. Reuse message objects instead of creating new ones per operation
2. Use the appropriate backend (upb for Python, C++ runtime for JVM)
3. Consider the lite runtime for mobile/embedded
4. Use ByteString (Java) to avoid copying byte arrays

Q: Are protobuf messages thread-safe?

Immutable message objects (e.g., built Java messages) are thread-safe for reading. Mutable builders and Python message objects are NOT thread-safe — use external synchronization if sharing across threads.

Interoperability

Q: Can I mix proto2 and proto3 messages?

Yes, proto2 and proto3 messages can import and reference each other. However, some features like required fields (proto2-only) won't be available when accessed from proto3 code.

Q: How do I send protobuf over HTTP?

Serialize to bytes and send with Content-Type: application/x-protobuf (binary) or use JSON serialization with Content-Type: application/json.

Q: Does protobuf support streaming?

The wire format itself doesn't include message framing — you need to prepend length headers when sending multiple messages over a stream. Use writeDelimitedTo()/parseDelimitedFrom() (Java) or implement length-prefixed framing.

Q: Can I convert between proto2 and proto3?

Use the Editions migration tool (protoc --editions_migration) or manually convert. Proto3 drops required fields and custom defaults, so review those semantics before converting.

Build and Tooling

Q: Do I need Bazel to use protobuf?

No. Protobuf supports both Bazel and CMake for building from source. Most users only need the protoc binary (available as pre-built downloads) and a language-specific runtime installed via their package manager.

Q: How do I use protobuf with gRPC?

Define service blocks in your .proto files and use the gRPC protoc plugin for your language (e.g., --grpc-java_out, --grpc_python_out). The gRPC framework handles the RPC transport.

Q: Why does protoc not find my imported .proto files?

Use the -I (or --proto_path) flag to specify directories where protoc should search for imports. Multiple -I flags can be specified. The import path in the .proto file must be relative to one of the -I directories.

Q: How do I generate code for multiple languages at once?

Specify multiple output flags in a single protoc invocation:

protoc --java_out=./java --python_out=./python --cpp_out=./cpp myproto.proto

Q: Where are the well-known type .proto files?

They're bundled with the protoc binary in the include/ directory of the release archive. The compiler automatically finds them when you use -I to point at that directory, or when they're installed to a system include path.