Exchanging a number between services sounds simple until you need to agree on all its possible meanings. An inventory service might report twelve available units for a product, but what does zero mean? Does a missing value mean unknown? Can an older client read a response containing a newly added field? Those decisions matter as much as getting the bytes from one service to another.

Protocol Buffers provides a structured way to define and serialize messages across languages. Its usefulness depends on how carefully you design those messages.

This chapter explains field types, presence, binary encoding, and the compatibility decisions that keep an API usable as clients and servers change independently.

Protocol Buffers, or Protobuf, combines a schema language, a binary serialization format, and tooling for working with messages. A schema describes the fields a message can contain. A message is a particular instance of that structure, such as an inventory observation for one warehouse.

The schema normally lives in a .proto file. The protoc compiler generates language-specific message code from it. Application code populates those messages, and the Protobuf runtime serializes them into bytes or parses bytes it receives from elsewhere.

Protobuf does not send requests, authenticate callers, or query a database. gRPC can carry Protobuf messages, but applications can also use the same format in files or event payloads.

The following diagram separates the schema developers use during development from the data applications exchange at runtime:

The bytes do not normally carry the schema or field names. The reader needs the appropriate definition to interpret them meaningfully. Sharing message definitions is therefore part of maintaining the API.

The examples use syntax = "proto3";. Protobuf also supports proto2 and Editions. With Editions, the schema selects an edition and feature settings that control language behavior. Editions uses explicit presence by default for singular fields. Proto3 remains useful for illustrating the distinction between implicit and explicit presence; do not assume its defaults apply unchanged to Editions schemas.

Assume an internal API returns inventory observations to authorized services. Each record identifies a product and warehouse, reports available units, and may include when the service observed the stock. Reading a record does not reserve stock.

Save this schema as inventory.proto:

A declaration such as optional int32 available_quantity = 3; gives a field its presence behavior, type, name, and number. The number identifies the field in the binary format. It is neither a default value nor an array position.

Field numbers must be unique within their message. Once you deploy code that uses a message type, keep its field numbers stable. Rearranging declarations for readability is different from renumbering them: changing the number changes what readers recognize.

This schema introduces a distinct InventoryRecord message. Field numbers are local to that message; they do not need to match numbers in a separate request or response type.

A legitimate business record could have identifiers sku_1042 and wh_blr_01, an explicitly supplied quantity of 0, and an active state. That means the service tracks the item, which is currently out of stock. A record that supplies no quantity fails this API's contract, even though Protobuf can represent and serialize it.

The distinction between representable data and valid business data matters throughout message design.

Choose a type for the meaning and range of the value, rather than for its appearance in one example.

For money, define an exact representation with a currency and documented units or use an appropriate shared money type. A floating-point field named price leaves precision and currency decisions unresolved.

Collections and Alternatives

A repeated field represents a sequence, while a map represents key-value associations. The inventory record uses a sequence of fulfillment methods and a map of labels. Do not rely on map iteration or serialization order. If ordering is part of the contract, model it explicitly.

Keep labels for descriptive metadata. Putting an authorization scope, stock quantity, or other core business field into map<string, string> makes its type and validation rules less visible. Also specify limits on collection sizes and string lengths; the schema above does not enforce those limits.

An enum gives a field named numeric values. In proto3, its first value must be zero. An UNSPECIFIED zero value avoids treating an uninitialized value as a meaningful business state.

Plan for unfamiliar enum values as the API evolves. A future server might add a state that older code does not recognize. Clients need an explicit fallback, such as displaying an unknown state while declining to make a fulfillment decision from it. They must not silently interpret it as ACTIVE.

A oneof groups alternatives so that a generated message holds at most one member of the group. For example, a separate lookup message might accept either a product identifier or a barcode:

This is a standalone message excerpt for the same package. A oneof can also be unset, so it does not enforce “you must supply exactly one.” In generated APIs, setting one alternative clears the other. The service must still require a selection and validate the chosen value.

Shared Time Types

google.protobuf.Timestamp represents an instant using seconds and nanoseconds relative to the Unix epoch. google.protobuf.Duration represents a span of time. Using distinct types avoids confusing “observed at this instant” with “valid for this duration.”

The observed_at field still needs a business definition. Here it means when the service observed availability, not when it sent the response. A correctly encoded timestamp with the wrong meaning can be just as misleading as an invalid one.

Field presence means knowing whether the sender explicitly supplied a field, separately from reading its value.

For a proto3 scalar without optional, an omitted field and an explicitly assigned default are indistinguishable through the usual generated API. Numeric defaults are zero, strings default to empty, and booleans default to false. The serializer normally omits those implicit default values.

The inventory schema uses optional so that the application can distinguish an absent quantity from an explicit zero. Reading an unset quantity still returns zero; the application must check presence to tell the cases apart.

The diagram shows this API's validation policy after parsing:

Presence preserves information the application needs for the decision. It does not make the decision automatically.

With protoc, the Python Protobuf runtime, and the standard Protobuf include files installed, generate message code from the schema:

If the compiler installation does not automatically locate standard imports, add its include directory with another --proto_path option. This command generates message code only; you do not need a gRPC plugin.

The following Python example uses that generated module:

The value is zero in both cases, but the explicit zero survives serialization with its presence intact. The example isolates presence behavior; a production record must satisfy the API's other validation rules too.

Singular message fields such as observed_at already track presence in proto3 without optional. Repeated fields and maps do not distinguish absence from an empty collection. If that difference matters, use a deliberately designed wrapper message or update contract.

The keyword optional describes representation, not business permission to omit a value. This API requires both identifiers and quantity. A service accepting these records must check those requirements, reject empty identifiers and negative quantities, and enforce collection limits. It must separately authorize access to the warehouse; a valid message proves nothing about the caller's permissions.

A serialized field carries a tag combining its field number and wire type, followed by its encoded value. The wire type tells the parser how to read or skip the value. It does not supply the field's business meaning or its full declared type.

For a focused encoding example, take an InventoryRecord with only available_quantity = 12. This is a serialization demonstration, not a complete business-valid record. The resulting bytes are:

These are hexadecimal bytes. Field number 3 uses wire type 0, so its tag is (3 << 3) | 0, which equals decimal 24, or hexadecimal 18. The encoder represents twelve as 0c.

Integer values can use a varint, a variable-length encoding in which small values take fewer bytes. Strings, embedded messages, and byte sequences use length-delimited encoding. This lets a parser skip a field it does not recognize.

Compact tags and omitted fields can reduce payload size, but Protobuf is not compression or encryption. The two bytes above contain neither gRPC framing nor network headers, so they are not the total cost of sending a call. Compare complete exchanges for performance decisions.

Serialization is also not canonical: logically equivalent messages can produce different bytes across implementations or builds, even with deterministic serialization options. If different systems must produce the same hash for equivalent messages, first define a canonical representation. Do not assume raw Protobuf bytes provide one.

During a rolling deployment, old and new readers coexist. Compatibility requires checking both whether readers can parse the bytes and whether the resulting behavior remains correct.

Suppose the inventory service adds an optional explanatory note. The updated InventoryRecord gains this field:

An older binary reader does not interpret field 8; a newer reader receiving an older message sees the field as absent. That makes the addition useful only if absence has a safe meaning. In this example, absence means no explanatory note is available, while the existing quantity keeps its meaning.

The diagram shows both directions of that exchange:

Neither exchange requires every reader to understand the note. By contrast, adding a request field that the server immediately requires can break older clients even when their messages still parse.

Use these distinctions during schema review:

When removing a field after consumers have migrated, reserve its number and name. For example, if you retire labels, remove its declaration and add these statements inside InventoryRecord:

Do not leave the old declaration alongside the reservation. Reserving prevents accidental reuse in future schema edits; it does not make removing behavior safe for callers still depending on it. Apply the same discipline to retired enum values.

Proto3 binary parsing preserves unknown fields for reserialization. However, converting through JSON or rebuilding a message by copying only known fields can lose them. This matters when an older intermediary forwards a newer message: test the actual forwarding path, not just direct communication between producer and consumer.

Adding optional to an existing implicit scalar also deserves care. The binary field remains compatible, but an older intermediary using implicit presence can drop an explicit zero when it parses and reserializes the message. End-to-end presence guarantees require every relevant hop to preserve that information.

The animation below shows how changes to a Protobuf schema affect old and new readers.

ProtoJSON is Protobuf's defined JSON mapping. It is useful at system boundaries, but it has different compatibility properties from the binary format.

By default, field names become lower camel case, enum values use their names, and the serializer emits 64-bit integers as decimal strings. Bytes use Base64. A timestamp uses a formatted time string rather than an object containing seconds and nanoseconds.

For example, a ProtoJSON representation of a populated inventory record could be:

This is a JSON representation of the message, not the bytes carried by a normal binary Protobuf exchange. Use the Protobuf JSON library rather than serializing generated objects with an arbitrary JSON serializer.

JSON contains field and enum names, so renaming them can affect consumers. ProtoJSON also does not preserve unknown fields, and parsers reject unknown fields by default unless you configure the parsers to ignore them. A binary-safe addition is therefore not automatically safe for a deployed JSON path.

Tooling versions introduce another boundary. Wire compatibility between messages is separate from compatibility between generated code and its runtime library. Pin the compiler, relevant plugins, and runtime dependencies to supported combinations. Regenerate code through the build workflow rather than editing generated files.

For a consequential schema change, test an old reader with new data, a new reader with old data, and any intermediary or JSON conversion involved. Check meaningful edge cases: explicit zero, omitted optional values, unfamiliar enum values, and missing newly added fields. A schema that compiles has passed a syntax check, not a complete compatibility review.

Protocol Buffers turns a shared message definition into language-specific code and a compact binary representation. Field numbers identify data on the wire, while field types, presence, and application validation determine how applications can use that data.

Design absence and defaults deliberately, preserve deployed field identities, and give new fields safe behavior for older participants. Review binary, generated-code, JSON, and business compatibility separately. Successful parsing is essential, but it does not by itself establish a correct or authorized API operation.