Skip to content

CONCEPT Cited by 1 source

MemoryOffsetBackingStore (Debezium)

Definition

MemoryOffsetBackingStore is a Kafka Connect offset-store implementation used by Debezium (and embeddable via Debezium Engine) that holds connector offsets in JVM heap memory only — offsets do not persist across connector restarts. On every restart, the connector has no prior offset and must resume from whatever position the upstream source reports (for Postgres, the replication slot's confirmed_flush_lsn).

Structurally opposite to the default Kafka Connect offset-store shape (compacted Kafka topic, persistent, durable).

Why Zalando uses it

Zalando's 2025-12 post canonicalises the choice verbatim:

"Since we launched Fabric Event Streams in late 2018, we've always treated the PostgreSQL replication slot as the authoritative source of truth for stream position. Because we didn't care what offset Debezium tracked internally, we ran with the ephemeral MemoryOffsetBackingStore, meaning our connectors always deferred to the slot position on startup and the keepalive flush advancing the slot ahead of the stored offset was never a problem for us." (Source: sources/2025-12-18-zalando-contributing-to-debezium-fixing-logical-replication-at-scale)

The choice is architectural, not operational convenience:

  • Ephemeral offset store → slot is always authoritative on startup.
  • No second-location mismatch possible (the slot-vs-offset mismatch problem is structurally precluded).
  • pgjdbc keepalive flush is always safe — the driver advances the slot, the next connector restart reads from the slot, no "offset is behind slot" conflict.

The required substrate

The choice only works when the replication slot is durable across primary failover. Zalando enables this via:

  • Patroni-managed Postgres HA since the mid-2010s.
  • Postgres Operator for Kubernetes orchestration.
  • Replication-slot management that ensures slots survive failovers from day one of their logical-replication rollout in late 2018.

Without slot-survives-failover discipline, ephemeral offset store → any primary failover or slot recreation → connector has no prior state to fall back to → full re-snapshot of the source database. This is why most Debezium users don't use MemoryOffsetBackingStore: Kafka Connect offset topics provide durability the underlying Postgres slot doesn't.

Trade-off axes

Axis MemoryOffsetBackingStore External offset store (Kafka Connect)
Slot-vs-offset mismatch Impossible by construction Possible and must be resolved on startup
pgjdbc keepalive flush Always safe Can break offset-store contract
Slot failover requirement Hard requirement Soft (offset store covers gaps)
Recovery on slot loss Full re-snapshot Streams from stored offset if slot is recreated with compatible position
Operational complexity Low (no offset-store infra) Moderate (Kafka Connect offset topics)

Seen in

  • sources/2025-12-18-zalando-contributing-to-debezium-fixing-logical-replication-at-scale — canonical wiki introduction. Zalando's 2025-12 post canonicalises MemoryOffsetBackingStore as the load-bearing architectural choice that makes Zalando's deployment "apparently the only ones who liked" the pgjdbc keepalive-flush feature. The choice pairs with Patroni-managed slot-survives-failover discipline to make the replication slot the single authoritative position-tracking source.
Last updated · 428 distilled / 1,221 read