Skip to content

CONCEPT Cited by 1 source

Proto3 explicit optional

Definition

Protobuf 3's three-way history with field presence: proto2 had required and optional labels; proto3 (initial) dropped both — every field was implicitly "optional with default," but presence was not introspectable for singular scalars; proto3 3.15 (February 2021) reintroduced the optional label on singular scalars to restore HasField() / .has_X presence checks. The three solutions (from most to least preferred for new work):

  1. optional label (proto3 ≥ 3.15): 
    message EventDataC {
    optional uint32 payload_size_bytes = 1;
}
  2. Wrapper types (pre-3.15 fallback, still useful where upgrade is expensive): 
    import "google/protobuf/wrappers.proto";
message EventDataC {
    google.protobuf.UInt32Value payload_size_bytes = 1;
}
  3. oneof hack (one-field "oneof" trick): 
    message EventDataC {
    oneof _payload_size_bytes_presence {
        uint32 payload_size_bytes = 1;
    }
}

Why this problem exists

In proto3's initial design, scalar fields always initialise to their type's default on the consumer — 0 for numerics, "" for strings, false for bool, empty for repeated. A producer that left a field unset and a producer that set it to 0 generate identical wire bytes. On the consumer:

if event_pb.payload_size_bytes == 0:   # ⚠️ ambiguous
    # was it absent, or was it actually 0?

This ambiguity breaks multiple common schemas:

  • Counters where 0 is a legitimate value distinct from "no reading."
  • Timestamps before the Unix epoch (= negative, but consumers often compare against 0 for "unset").
  • Flags where false and "not applicable" differ.
  • Updates — patch semantics where absent means "don't change."

The proto2 required label attempted to prevent this class of bug, but "it was nearly impossible to safely change a required field to be optional" without breaking every old client. Removing required cleaned up that migration cliff but introduced the presence ambiguity — which 3.15's optional revival addresses.

Timeline

Version Date Behaviour
proto2 2008 required, optional, repeated labels. Presence available on optionals.
proto3 2016 required and optional removed. Every singular field is "optional." No presence on scalars.
proto3 2021-02 (3.15) optional label reintroduced on singular scalars. HasField() works again.

Enums in proto3 added a related problem that the UNKNOWN = 0 convention addresses — see concepts/unknown-zero-enum-value.

Proto2 required was a one-way door

Per the 2024-09-16 Lyft post:

"The required label was enforced strictly by the compiler which proved hugely problematic in the long run, because it was nearly impossible to safely change a required field to be optional."

If a proto2 producer marks X required and ships it, every consumer depends on X being present. Relaxing X to optional breaks every consumer's parse. The migration is unbounded-clients × unbounded-time. Google concluded the right answer was to remove the label from the language — presence semantics should be modelled via oneof (tagged union) or wrapper types.

Using .HasField()

After optional:

# proto3 ≥ 3.15
if not event_pb.data_c.HasField('payload_size_bytes'):
    handle_absent()
elif event_pb.data_c.payload_size_bytes == 0:
    handle_zero()
else:
    handle_nonzero(event_pb.data_c.payload_size_bytes)

The distinction between "absent" and "zero" is now representable.

Lyft's convention

Lyft Media adopted protobuf before 3.15 existed, so their legacy convention is to use google.protobuf.*Value wrappers:

"Since protobufs were adopted at Lyft prior to the introduction of optionals to the language specification, our convention for optional primitive types is to use wrappers from the google.protobuf package."

Both wrapper types and the optional label produce the same observable HasField semantics; wrappers cost slightly more bytes on the wire (an extra nested-message header) but carry forward cleanly without a schema migration.

Seen in

  • sources/2024-09-16-lyft-protocol-buffer-design-principles-and-practices — canonical statement on the wiki. Lyft Media's post walks through the historical context (proto2 required → proto3 drop → 3.15 reintroduction), names the bug (absent vs default indistinguishable), shows both fixes (optional label + wrapper types), and discloses Lyft's specific choice to standardise on wrappers for pre-3.15 codebases.
Last updated · 319 distilled / 1,201 read