SYSTEM Cited by 1 source
Minerva (eBay)¶
Minerva is eBay's observability platform for its Monetization systems (fees, pricing, promotions, subscriptions). It gives operators a shared, timely view of fee behaviour and the business context behind it without adding latency or availability risk to the services that calculate and charge fees. Its defining design choice is a strict separation between making a fee decision (online, on the critical path) and explaining the resulting behaviour (asynchronous, off the critical path).
Why it exists¶
Before Minerva, eBay's Monetization team had no dedicated observability platform. Transactional data in the datawarehouse (HDFS) arrived too late for timely launch monitoring (and often sat in a different security zone), investigations required engineers to query multiple sources and assemble comparisons by hand, and unexpected fee behaviour frequently surfaced only through customer complaints or post-transaction analysis — a classic grey-failure gap where a global total looks normal while one site, fee type, promotion, or time window misbehaves. At eBay's scale (Q2 2026: $22.4B GMV, 136M active buyers, 2.6B active listings; pricing driven by ~200 item/seller/marketplace/campaign attributes), aggregate correctness is not enough — monitoring must preserve enough dimensions to answer both what changed and why.
Architecture¶
End-to-end, Minerva is an asynchronous observability pipeline hanging off the fee-processing service:
- Local capture (hot path). The fee-processing service records the fee decision and places a tiny identifier-only record (item + fee-calculation event IDs) on a local in-process queue. Under normal conditions the request does not wait for network publish, enrichment, analytical storage, or alert evaluation — the hot path touches only the local queue. This is the data-plane/control-plane split applied to monitoring.
- Bounded local queue (ring buffer). The queue absorbs short-lived downstream delays but does not eliminate backpressure: it is bounded by capacity and record age, and records that cannot be processed within a configured interval are flushed or overwritten (ring-buffer policy) rather than allowed to grow unbounded. The explicit trade is fee-processing availability over lossless monitoring delivery — a prolonged downstream interruption creates a gap in monitoring data but can never change or invalidate the original fee calculation.
- Publish + async enrichment. A background producer drains the queue to an event stream. Because the record carries only identifiers, a background consumer queries the transaction read database to retrieve fee, listing, account, and promotion attributes, producing enriched records that serve dashboards without repeated joins across operational stores. Monitoring is therefore eventually consistent by design — data arrives after the fee decision.
- Analytical storage. Batches of enriched records land in ClickHouse. Common queries (fee trends / promotion usage by site / category / user) aggregate a small set of fields — fee amount, site, fee type, promotion, time — across millions of records each with almost 250 columns; ClickHouse's columnar layout reads only the needed fields and its MergeTree engines align ordering/partitioning to monitoring access patterns. For seller/item analyses, the design considered placing related records on the same shard to reduce cross-node (scatter-gather) work.
- Signal generation (three modules). Over the stored history: query execution (identify eligible queries, check cadence, and — because multiple monitoring instances run concurrently — coordinate so only one instance evaluates each scheduled query; multi-query use cases wait for the required result set); use-case processing (per-signal logic — fee-amount deviation, transaction-count-by-site-and-hour comparison, combining fee + promotion data to interpret a spike — assigns severity and builds a notification model with context); and notification delivery (channel-specific payloads for PagerDuty / email / Slack, with settings defined broadly then overridden per rule or data segment, e.g. high-severity country-level deviation → PagerDuty vs a lower-severity trend → email). This is the severity-gated routing discipline.
The identifier-only hot-path record projected into an enriched, query-optimized ClickHouse store is a concrete instance of the async-projected read model.
Design stance¶
- Observe, don't recompute. Minerva consumes upstream fee and promotion decisions rather than reimplementing their rules — "observe monetization outcomes, not become a second fee-calculation engine."
- Monitoring is not the source of truth. The event stream is a lossy, time-windowed monitoring substrate; the transactional fee systems remain authoritative.
- Extensibility via separation. New monitoring use cases (queries + processing + notifications) can be added without changing the fee-processing service or the ingestion pipeline.
Numbers (as reported)¶
- Feasibility PoC: ~1.1B listing-session rows on 6 vCPU / 44 GB, sample lookups/aggregations ~0.11–0.72 s (< 1 s) — the article flags this as not a production benchmark (no repeated runs, cache state, concurrency, compression, or baseline recorded).
- Enriched records: ~250 columns each, aggregated across millions of rows.