CONCEPT Cited by 1 source

Last Reconciled Offset

Definition

The Last Reconciled Offset is a per-partition watermark tracked by the Redpanda Cloud Topics architecture that marks the boundary between offsets whose data still lives only in L0 files (ingest-optimised, cross- partition) and offsets whose data has been reorganised into L1 files (read-optimised, per-partition, offset-sorted) by the background Reconciler.

It is the minimum coordination point between the write path (which only appends to L0 and the partition's Raft log of placeholder batches) and the read path (which branches at this one boundary) — a single monotonic counter per partition that eliminates the need for the reader to enumerate files.

Canonical wiki instance

Introduced from the 2026-03-30 Redpanda Cloud Topics architecture deep-dive:

"When a consumer requests data, Redpanda routes the request based on where the data currently lives in its lifecycle. Each partition tracks a Last Reconciled Offset.

• Reads > Last Reconciled Offset: The system reads from L0. The system follows the pointers in the local Raft logs to find the specific batches in object storage if not found in the local cache.
• Reads < Last Reconciled Offset: The system reads from L1. This is the highly optimized path for historical reads, allowing us to open large, sorted files and stream data efficiently without scattering."

Why a single watermark is sufficient

The Reconciler processes partitions in offset order: it reads L0 files containing offsets [a, b] for partition P, writes an L1 file covering [a, b], and advances partition P's Last Reconciled Offset to b. This monotonic advance is what collapses the routing decision to one integer comparison per consumer fetch: if the requested offset is below the watermark, L1 has it; if above, L0 has it (or the memory cache ahead of L0).

No per-offset index, no per-file bookmark, no per-segment map on the read path — just compare requested_offset to last_reconciled_offset for that partition.

Lifecycle properties

• Per-partition — each partition has its own watermark; no cluster-wide or topic-wide coordination.
• Monotonic advance — the Reconciler only ever moves it forward as L0 → L1 rewrites commit.
• Minimum coordination point — write path and read path communicate only through this one boundary; neither needs knowledge of the Reconciler's internal schedule.
• Metadata side effect — updating the watermark is part of the L1-metadata-tier commit ("backed by an internal topic and a key-value store"), not a Raft log append on the hot write path.

Read-path behaviour at the boundary

requested_offset > last_reconciled_offset → L0 / cache path:

1. Try the per-broker memory cache (fast path for tailing consumers).
2. On cache miss, follow the placeholder batch in the partition's Raft log to the L0 object-storage location; issue GET.

requested_offset < last_reconciled_offset → L1 path:

1. Look up the relevant L1 file in the shared L1 metadata tier (internal topic + KV store).
2. Open the file; seek to offset (offset-sorted); stream contiguous bytes.

This split means the scattered-read cost of L0 (see concepts/l0-l1-file-compaction-for-object-store-streaming) only applies to the small window between the memory cache's trailing edge and the Reconciler's trailing edge. As long as Reconciler cadence keeps pace with producer throughput, this window is bounded — making the average historical-read cost closer to L1's sorted-stream cost than L0's scattered-read cost.

Contrast with Kafka's high-water-mark

Kafka's high-water-mark is the highest offset replicated to all in-sync replicas, used to gate consumer-visible reads and anchor consumer offset commits. Last Reconciled Offset is a different axis:

Watermark	What it gates	Moves on	Granularity
Kafka high-water-mark	Consumer visibility	ISR replication	Per-partition
Last Reconciled Offset	Read routing (L0 vs L1)	Background Reconciler	Per-partition
Kafka log-end-offset	Producer append position	Every produce	Per-partition

They compose: the high-water-mark determines whether a record is readable; the Last Reconciled Offset determines where a readable record's data lives. A read can only succeed if requested_offset ≤ high_water_mark; the L0/L1 branch is chosen by requested_offset vs last_reconciled_offset.

Seen in

• sources/2026-03-30-redpanda-under-the-hood-redpanda-cloud-topics-architecture — canonical wiki source introducing the Last Reconciled Offset as Cloud Topics' read-routing watermark.

Related

• systems/redpanda-cloud-topics — the system that uses this watermark.
• systems/redpanda — the broker.
• concepts/l0-l1-file-compaction-for-object-store-streaming — the two-tier layout this watermark routes between.
• concepts/placeholder-batch-metadata-in-raft — the L0-side metadata the watermark boundary points to.
• concepts/kafka-consumer-lag-metric — one common use of watermarks; Last Reconciled Offset is a different watermark with a different job.
• concepts/high-water-mark — the established Kafka watermark this one composes with.
• concepts/offset-commit-cost — unrelated axis; Last Reconciled Offset is not stored per-consumer-group.
• patterns/background-reconciler-for-read-path-optimization — the broader pattern whose coordination is entirely compressed into this single scalar.