The magic of Protobuf binary encoding isn’t just its size efficiency, it’s how it packs multiple distinct data types into a single, contiguous byte stream without any explicit delimiters.
Let’s see what a simple Protobuf message looks like when it hits the wire.
Imagine this .proto definition:
message Person {
string name = 1;
int32 id = 2;
bool is_employed = 3;
}
And we want to serialize an instance like this: name: "Alice", id: 123, is_employed: true.
The serialized output, when viewed as hex, might look something like this:
08 75 12 05 41 6c 69 63 65 18 01
Let’s break this down. Protobuf’s binary format is built around "key-value" pairs, where the "key" is actually a tag. This tag is a small integer that uniquely identifies a field within a message type. The tag is encoded as a varint, which is a variable-length encoding for integers.
The first byte, 08, is our first tag. To decode this, we look at the least significant 3 bits. If they are 000, it’s a varint. If they are 001, it’s a 64-bit value, and so on. In 08 (binary 00001000), the last 3 bits are 000, indicating a varint. The remaining bits 00010 are shifted left by 3 bits to form the field number, resulting in 2. This 2 corresponds to the id field in our .proto definition (because id = 2). The value 75 (hex) is the varint-encoded value of our id, which is 123 in decimal. So, 08 75 represents id: 123.
The next byte, 12, is our second tag. Again, the last 3 bits are 000, indicating a varint. The remaining bits 0010 shifted left by 3 bits give us 4, which corresponds to the name field (because name = 1). The rest of the bytes 05 41 6c 69 63 65 represent the value. The 05 tells us the length of the following string in bytes. The subsequent bytes 41 6c 69 63 65 are the ASCII (or UTF-8) representation of "Alice". So, 12 05 41 6c 69 63 65 represents name: "Alice".
Finally, 18 is our third tag. The last 3 bits are 000, indicating a varint. The remaining bits 0011 shifted left by 3 bits give us 6, which corresponds to the is_employed field (because is_employed = 3). The value 01 (hex) is the varint-encoded boolean true. So, 18 01 represents is_employed: true.
Putting it all together: 08 75 (id: 123), 12 05 41 6c 69 63 65 (name: "Alice"), 18 01 (is_employed: true). This is how Protobuf achieves its compact binary format, by encoding the field tag and its value sequentially, using varints for efficiency where appropriate.
The key insight here is that the field number isn’t directly encoded as the tag; it’s derived. The wire type (the last 3 bits of the tag byte) tells the parser how to interpret the subsequent bytes. For varints, it’s a direct read. For strings and byte arrays, it’s a length-prefixed read. For packed repeated fields, it’s a sequence of values all encoded with their respective wire types.
The fact that the field number is explicitly encoded in each tag means that you can add new fields to your .proto file without breaking older code that doesn’t know about them, as long as you don’t reuse field numbers. The old code will simply ignore the tags it doesn’t recognize.
The next step in understanding Protobuf’s wire format is to explore how repeated fields, especially packed ones, are encoded differently.