EBAY 2026-10-14 Tier 3

Inside Minerva: building scalable observability for eBay's monetization platform¶

Summary¶

Minerva is eBay's observability platform for its Monetization (fee / pricing / promotion / subscription) systems. The core problem: a sudden change in fee activity can be a pricing defect or an expected effect of a promotion ending, a marketplace fee-structure change, or shifting customer behaviour — and distinguishing them requires timely historical data plus the ~200 item/seller/marketplace/campaign attributes behind each fee decision, not just aggregate totals. The design's central contribution is a strict separation between making a fee decision (online, on the critical path) and explaining the resulting behaviour (asynchronous, off the critical path): the fee-processing service writes a tiny identifier-only record to a local bounded queue, a background producer drains it to an event stream, an asynchronous consumer enriches it from transactional read databases, batches of enriched records land in ClickHouse, and reusable dashboards + monitoring use cases evaluate the history — so monitoring can never add latency or availability risk to the service that calculates and charges fees.

Key takeaways¶

Observability must stay off the fee-processing critical path. Publishing directly from a fee request to a remote event stream would couple monitoring availability to fee-processing availability (a slow network call, unavailable broker, or downstream backpressure would raise request latency or starve production traffic). Minerva instead places a small record — only the item + fee-calculation event identifiers — on a local in-process queue; under normal conditions the request never waits for network publish, enrichment, analytical storage, or alert evaluation. (Source: this article; the data-plane/control-plane split applied to monitoring.)
The local queue is bounded by capacity and record age (ring-buffer policy). The queue absorbs short downstream delays but does not eliminate backpressure: if the background producer cannot drain it for an extended period the backlog would eventually consume resources and affect the fee service, so records that cannot be processed within a configured interval are flushed or overwritten rather than allowed to grow unbounded. This deliberately prioritizes fee-processing availability over lossless monitoring delivery — a prolonged downstream interruption creates a gap in monitoring data but can never change or invalidate the original fee calculation (the event stream is not the transactional source of truth).
Asynchronous enrichment makes monitoring eventually consistent. Because the initial record carries only identifiers, a background consumer later queries the transaction read database to retrieve the fee, listing, account, and promotion attributes, producing enriched records that serve dashboards without repeated joins across operational stores. The data becomes available after the fee decision; the request stays isolated from the analytical work (eventual consistency as an intentional trade).
ClickHouse is the analytical store because transactional DBs can't take repeated historical scans. Common Minerva queries (fee trends and promotion usage by site / category / user) aggregate a small set of fields — fee amount, site, fee type, promotion, time — across millions of records each with almost 250 columns. ClickHouse's columnar layout reads only the needed fields, and its MergeTree engines provide ordering/partitioning aligned to monitoring access patterns. (Source: this article.)
Early feasibility PoC, explicitly not a production benchmark. ~1.1 billion listing-session rows on a 6-vCPU / 44 GB machine returned sample lookups and aggregations in < 1 s (reported examples ~0.11–0.72 s). The notes do not record repeated runs, cache state, concurrency, compression, or a comparison baseline — a feasibility signal, not a benchmark.
Data modeling is a monetization-specific trade-off. A listing can have a base fee plus multiple promotional adjustments: flattening those values simplifies scans but duplicates context; joining normalized records avoids duplication but adds query-time work. The first model organized records around listing-session and account-event identifiers, and for seller/item analyses considered placing related records on the same shard to reduce cross-node work. Minerva consumes upstream fee/promotion decisions — "observe monetization outcomes, not become a second fee-calculation engine."
Signal generation is a three-module pipeline over the stored history. (1) Query execution identifies eligible queries and checks cadence; because multiple monitoring instances run simultaneously they coordinate so only one instance evaluates each scheduled query, and a use case that depends on several queries waits until the required result set is available. (2) Use-case processing applies the per-signal logic (fee-amount deviation; transaction-count-by-site-and-hour comparison; combining fee + promotion data to interpret a spike), assigns severity, and builds a notification model with supporting context. (3) Notification delivery builds a channel-specific payload for PagerDuty / email / Slack, with settings definable broadly then overridden per rule or data segment (e.g. a high-severity country-level fee deviation → PagerDuty, a lower-severity trend → email) — the severity-gated routing discipline.
The separation is the extensibility story. New monitoring use cases (queries + processing logic + notifications) can be added without changing the fee-processing service or the ingestion pipeline — the platform evolves alongside eBay's monetization systems because explaining behaviour is decoupled from making the decision.

Systems¶

Minerva — eBay's monetization observability platform (primary subject): local-queue capture → async enrichment → ClickHouse history → reusable dashboards + three-module monitoring pipeline.
ClickHouse — the columnar/MergeTree analytical store holding the enriched fee history (~250-column records, millions of rows).

Concepts & patterns¶

concepts/control-plane-data-plane-separation — monitoring (control/observation plane) is kept strictly off the fee-processing data plane; the local record is the only hot-path touchpoint.
concepts/hot-path — the fee request writes only identifiers to a local queue; all heavy work (publish, enrich, store, evaluate) is off the hot path.
concepts/backpressure — bounded local queue with a ring-buffer flush/overwrite policy; availability prioritized over lossless delivery.
concepts/eventual-consistency — enrichment happens after the fee decision, so the monitoring view is eventually consistent by design.
concepts/columnar-storage-format — ClickHouse reads only the handful of fields a monitoring query needs out of ~250 columns.
concepts/scatter-gather-query — fee-trend/promotion aggregations fan across the ClickHouse analytical history; sharding is considered on seller/item to reduce cross-node work.
concepts/alert-fatigue — per-rule/per-segment severity routing (PagerDuty vs email vs Slack) and single-instance query de-duplication keep signals actionable.
concepts/grey-failure — the motivating failure class: a global total looks normal while one site / fee type / promotion / time window behaves unexpectedly, previously surfacing only via customer complaints or post-transaction analysis.
patterns/async-projected-read-model — the identifier-only hot-path record is asynchronously projected into an enriched, query-optimized ClickHouse read model.
tiered-storage / monitoring-is-not-source-of-truth — the event stream is a lossy, time-windowed monitoring substrate, explicitly not the transactional record of fee decisions (recorded as prose; the authoritative state lives in the fee systems).

Operational numbers¶

eBay Q2 2026 scale context: $22.4B GMV, 136M active buyers, 2.6B active listings; pricing depends on ~200 item/seller/marketplace/campaign attributes.
Enriched records: almost 250 columns each, aggregated across millions of rows.
PoC: ~1.1B listing-session rows on 6 vCPU / 44 GB, sample queries ~0.11–0.72 s (< 1 s) — feasibility, not a production benchmark.

Caveats¶

Tier-3 eBay source; included because the body is genuine distributed-systems architecture (hot-path decoupling, bounded local queue + ring-buffer backpressure, async enrichment, columnar analytical store, single-instance query coordination, severity-gated alerting) well above the Tier-3 bar — not seller/buyer-feature PR.
The only quantitative figures are the business-scale context and an early PoC the article itself flags as not a benchmark (no repeated runs, cache state, concurrency, compression, or baseline). No production latency / throughput / backlog-depth numbers.
Data-model details (flatten-vs-normalize, same-shard colocation for seller/item) are described as design considerations of the first model, not a finalized production schema.
The event stream and enrichment source databases are named only generically ("event stream", "transaction read database") — no broker or store product is identified, so no system page is minted for them.