Skip to content

title: Lyft — Protocol Buffer Design: Principles and Practices for Collaborative Development type: source created: 2026-04-22 updated: 2026-04-22 tier: 2 company: lyft published: 2024-09-16 url: https://eng.lyft.com/protocol-buffer-design-principles-and-practices-for-collaborative-development-8f5aa7e6ed85 raw: raw/lyft/2024-09-16-protocol-buffer-design-principles-and-practices-for-collabor-1e54ad9c.md tags: [protobuf, protocol-buffers, schema-design, api-design, proto3, validation, oneof, enum, backward-compatibility, lyft, lyft-media] systems: [protobuf, protoc-gen-validate, grpc] concepts: [unknown-zero-enum-value, unit-suffix-field-naming, proto3-explicit-optional, clarity-over-efficiency-in-protocol-design, extensibility-protocol-design, backward-compatibility, schema-evolution] patterns: [oneof-over-enum-plus-field, protobuf-validation-rules, protobuf-cross-entity-constants] related: [systems/protobuf, systems/protoc-gen-validate, systems/grpc, concepts/unknown-zero-enum-value, concepts/unit-suffix-field-naming, concepts/proto3-explicit-optional, concepts/clarity-over-efficiency-in-protocol-design, concepts/extensibility-protocol-design, concepts/backward-compatibility, concepts/schema-evolution, patterns/oneof-over-enum-plus-field, patterns/protobuf-validation-rules, patterns/protobuf-cross-entity-constants]

Lyft — Protocol Buffer Design: Principles and Practices for Collaborative Development

Summary

Roman Kotenko (Lyft Media) distils Lyft Media's operational experience designing shared proto3 protobuf schemas across mobile (iOS / Android) and backend (Python) teams into a reusable protocol-design playbook. Two named principles — clarity and extensibility — frame five concrete practices: reserve 0 for an UNKNOWN enum value; prefer oneof over "discriminator enum plus type-specific optional field"; name fields with their unit (payload_size_bytes, timestamp_ms_utc) and use well-known types (google.protobuf.Timestamp, Duration, StringValue) rather than raw primitives; use the optional label (proto3 ≥ 3.15, 2021) or google.protobuf.*Value wrappers so HasField() distinguishes absent from default; declare validation constraints inline in the .proto with protoc-gen-validate (or its stable successor protovalidate) — noting that generated validators must be invoked explicitly, they do not run automatically on parse. Closing pattern: use custom EnumValueOptions extensions to attach cross-entity constants to enum values (mobile client and server stay aligned on #tag1 / #tag2 literals through the schema rather than by parallel code).

Key takeaways

  • Two load-bearing principles for protocol design: clarity + extensibility. "A well-designed protocol should define its messages in a way where it's not only explicit about which fields must be set. This prevents missetting any of the messages during implementation. In other words, good protocols leave no ambiguity for its implementers." Extensibility: "protocol structure is built with future vision and potential roadmap in mind … some foreseeable additions and breaking changes can be accounted for in advance." Protocol design puts greater emphasis on these than ordinary code because the cost of a breaking change is paid across every compiled client. (Source: sources/2024-09-16-lyft-protocol-buffer-design-principles-and-practices) Canonical concepts/clarity-over-efficiency-in-protocol-design + concepts/extensibility-protocol-design.

  • Prefer oneof over "discriminator enum + type-specific optional field". The naive shape — enum Kind { A, B, C } plus per-kind optional fields — is implicit about which combinations are valid. If kind == C, must payload_size be set? What if kind == A? Every new event-type added compounds the convoluted validation logic callers must maintain. oneof makes the discriminator and the payload the same field; exactly one of the branches can be set; unset branches don't serialise (wire-size win). Canonical patterns/oneof-over-enum-plus-field.

  • Reserve 0 as UNKNOWN for every enum. Old implementations that pre-date a new enum value receive new values as 0; if 0 is a legitimate value, the pre-existing and new consumers silently disagree. Making ENUM_UNKNOWN = 0 the first entry means the "before-this-addition" behaviour is explicit and greppable. Canonical concepts/unknown-zero-enum-value.

  • Name fields with their unit / semantics, not the raw primitive type. payload_size_bytes, not payload_size. timestamp_ms_utc, not timestamp. Even better: use google.protobuf.Timestamp and Duration well-known types so the semantics are enforced by the type system instead of naming convention. For team-specific recurring shapes (e.g. lat/lng), promote to a shared internal well-known type. Canonical concepts/unit-suffix-field-naming.

  • Proto3 deprecated required for backward-compat reasons. In proto2, a required field in a message couldn't safely be relaxed to optional without breaking every old producer — the required label was "enforced strictly by the compiler which proved hugely problematic in the long run." Proto3 dropped required entirely; every field is effectively optional and scalars initialise to their type's default value. Canonical concepts/proto3-explicit-optional.

  • Use the optional label (proto3 ≥ 3.15, 2021) or wrapper types to distinguish absent from default. Before the optional label was re-introduced in proto3, uint32 payload_size_bytes couldn't tell "0 bytes" from "unset" — both serialise as absent. With optional uint32 payload_size_bytes, generated code gains HasField('payload_size_bytes'). Lyft adopted protobufs before 3.15 so their legacy convention is to use google.protobuf.UInt32Value / StringValue wrappers instead; new work can use the optional label. Canonical concepts/proto3-explicit-optional.

  • Declare validation constraints inline in the .proto with protoc-gen-validate (PGV) or its stable successor protovalidate. Named rule families: (validate.required) = true on a oneof forces one of the branches to be set (default behaviour would accept none); (validate.rules).enum = { defined_only: true, not_in: [0] } closes the open-enum behaviour proto3 introduced and excludes UNKNOWN; (validate.rules).string = { min_len: 1, uuid: true, pattern: "..." } for format checks; (validate.rules).repeated = { min_len: 1, unique: true, items: {...} } for collection constraints. Critical caveat flagged verbatim — "if a message is formed in violation of the stated rules, nothing will fail until its validator is invoked" — validation is not automatic on parse; generated Validate() / protoc_gen_validate.validator.validate(msg) has to be called at every trust boundary. Canonical patterns/protobuf-validation-rules on systems/protoc-gen-validate.

  • Cross-entity constants via custom EnumValueOptions extensions. To align a mobile client and a server service on literal string values ("#tag1", "#tag2" etc.) without defining them twice in two codebases, declare a custom extension on google.protobuf.EnumValueOptions (the post picks a large-ish field number, 11117, "to avoid accidental collisions") and attach [(const_value) = "#tag1"] to each enum entry. Generated descriptor APIs let any language stack read the string back. Recommended only when the constants are stable and you control both deployment ends — otherwise ordinary enum dispatch is safer. Canonical patterns/protobuf-cross-entity-constants.

  • Language-dependent behaviours matter. Handling of absent vs default values, map entries with absent-value fields, namespace layout, and generated-accessor patterns differ across Python / Swift / Kotlin / Go / Java — the article names serialisation of default-valued map entries as a specific "some languages write them, some omit them" divergence. Teams shipping multi-language stacks must test cross-language round-trips rather than assume parity.

Key numbers

  • No production metrics disclosed — this is a design-practices post, not a rollout retrospective.
  • Named version: protobuf 3.15 (2021) as the release that reintroduced the optional field label on singular scalars in proto3.
  • Custom-option field-number guidance: "For a small project, picking an arbitrary large prime number should be safe enough." (Lyft's example uses 11117.)

Architecture notes

The article's final consolidated example:

syntax = "proto3";

import "google/protobuf/descriptor.proto";
import "google/protobuf/timestamp.proto";
import "validate/validate.proto";

extend google.protobuf.EnumValueOptions {
    string const_value = 11117;
}

enum EventTag {
    EVENT_TAG_UNKNOWN = 0 [(const_value) = ""];
    EVENT_TAG_1       = 1 [(const_value) = "#tag1"];
    EVENT_TAG_2       = 2 [(const_value) = "#tag2"];
}

message Event {
    string id = 1 [(validate.rules).string = {min_len: 1}];
    google.protobuf.Timestamp timestamp_utc = 2
        [(validate.rules).timestamp = {required: true}];
    oneof data_kind {
        option (validate.required) = true;
        EventDataA data_a = 3;
        EventDataB data_b = 4;
        EventDataC data_c = 5;
    }
}

message EventDataA {}
message EventDataB {}
message EventDataC {
    optional uint32 payload_size_bytes = 1;
}

Five design decisions are visible simultaneously: (1) 0 = UNKNOWN for EventTag; (2) cross-entity constants attached via [(const_value) = ...]; (3) unit-suffixed field names (timestamp_utc, payload_size_bytes); (4) oneof data_kind with option (validate.required) = true so exactly one branch must be set; (5) optional uint32 payload_size_bytes for presence semantics on a primitive. Well-known google.protobuf.Timestamp replaces a raw uint64. Validation rules are inline on each field and at the oneof.

Caveats

  • Design-practices essay, not a retrospective. No production scale, QPS, latency, or incident count. The post is a distillation of Lyft Media team conventions framed as general principles — the numbers side of this page is intentionally light.
  • Lyft Media context, not Lyft-wide. The practices describe Kotenko's team specifically; the 2020 cross-reference to Michael Rebello's "Lyft's Journey Through Mobile Networking" post confirms Lyft-wide adoption of protobufs for mobile-server traffic, but this post's conventions may not be universal across all Lyft orgs.
  • proto3-only. Proto2 content appears only as historical framing for why required was dropped.
  • PGV → protovalidate transition is flagged but not covered in depth. PGV is cited as stable, the migration cue points at protovalidate as the modern replacement, but validation examples throughout the post use PGV syntax; teams starting fresh should evaluate protovalidate directly.
  • Cross-entity constants via custom options is explicitly flagged by the author as "recommended to exercise caution … most suitable for cases where the constant values are never expected to change, or where you have complete control over deployment of entities that will be consuming the protocol" — a power-tool with enough rope to hang a team running heterogeneous clients.
  • No mobile-codebase specifics. The post mentions Python, Swift, Kotlin support abstractly but all code examples are Python; iOS and Android integration details are future-work.
  • Author's Medium-subscription CTA inserted mid-article — not architectural content, filtered out of the distillation.

Source

Last updated · 319 distilled / 1,201 read