PATTERN Cited by 1 source

Dual-stream telemetry pipeline¶

Pattern: emit observability data over two independent streams — a per-occurrence "notable" stream carrying full-fidelity data for the rare interesting tail (slow, heavy, or failing events) and a per-pattern aggregate stream carrying rolled-up statistics for the bulk of the workload. Each stream is tuned independently for volume, fidelity, retention, and downstream consumer shape. Avoids forcing the entire workload through either end of the detail/volume spectrum.

Problem¶

A single-stream observability pipeline must choose between:

Per-occurrence — full fidelity, but volume scales linearly with workload (millions of queries per second becomes millions of telemetry messages).
Fully aggregated — volume is bounded (one message per pattern per window), but the detail needed for debugging individual outlier occurrences (stack traces, query parameters, error messages, caller identity) is lost.

Neither end satisfies every downstream consumer: the debugger wants per-occurrence detail on a specific outlier; the capacity planner wants workload-level roll-ups with stable low volume; the anomaly detector wants aggregate time series.

Solution shape¶

Two streams, emitted concurrently from the same source:

Stream 1 — Notable / per-occurrence - Capture every event that crosses a tail threshold ("query took > 1s, read > 10k rows, or errored"). - One telemetry message per qualifying event. - Full fidelity: query text, caller tags, stack trace, error class. - Volume is bounded by the tail definition — typically a small fraction of total workload. - Consumers: debugging UIs, error-tracking tools, per- occurrence alerting.

Stream 2 — Aggregate / per-pattern - One message per query pattern per fixed window (e.g. 15s). - Counters: total executions, total runtime, rows-read, rows-returned, percentile latencies. - With per-unique-combination tagging (concepts/aggregate-tag-attribution): one message per distinct tag combination per pattern per window, bounded by dynamic cardinality reduction. - Volume bounded by pattern count × combinations × window frequency. - Consumers: query table, anomaly detection, trend graphs, cost attribution.

Each stream lands in its own storage backend (or topic, then a shared backend with different table schemas) tuned to its volume and query shape.

Canonical production instance¶

PlanetScale Insights (Source: sources/2026-04-21-planetscale-enhanced-tagging-in-postgres-query-insights):

Individual queries topic (Kafka): per-query message for any query reading > 10,000 rows OR taking > 1s OR returning an error. Powers the Notable queries feature (and the error-occurrences UI from the prior release).
Aggregate summaries topic (Kafka): per-query-pattern per-unique-tag-combination every 15s. Powers "the majority of Insights including the query table, anomalies, and all query-related graphs."
Both feed ClickHouse (see systems/clickhouse) as the storage backend, with separate table shapes.

Why this beats single-stream designs¶

Volume predictability: the aggregate stream is bounded by pattern count × combinations × window frequency, not by raw query volume. At scale, this is orders of magnitude smaller than per-query emission.
Fidelity where it's needed: the notable stream captures full context on the tail the debugger cares about; the cost is bounded because the tail is small.
Independent evolution: the two streams can be tuned independently — aggregate sampling policy, notable threshold adjustment, retention per stream, schema evolution per stream.
Storage-cost alignment: hot aggregate time-series lives in analytical engines (ClickHouse); notable occurrences can go to log-style storage with longer retention and different indexing.

Generalisation¶

The pattern is broadly applicable to any telemetry pipeline where workload volume is high and outlier frequency is low:

APM tracing: sample all traces (aggregate), capture full payload on error or slow traces (notable).
Log aggregation: aggregate log counts by structured fields, capture full log body on error/warn.
Metric rollups + raw samples: fast-aggregating histograms + exemplar capture for outlier values.
Query engines: statement-statistics rollups + slow-query logs (the Postgres pg_stat_statements + slow-log model is the archetype of this pattern at the per-query level).

The distinctive choice is making the two streams first- class and orthogonal, rather than deriving one from the other. Many systems treat the notable stream as "slow-log output from the aggregate pipeline" — but doing so entangles the schemas and couples the volume. Keeping them independent (same source, different sinks, different policies) maximises evolvability.

Relationship to tagging¶

Query tags (SQLCommenter) flow through both streams in the Insights architecture:

Notable stream carries tags per-occurrence (always — every notable query has its full tag set).
Aggregate stream carries tags per-unique-combination (subject to dynamic cardinality reduction).

Users filtering by tag (tag:actor:X) get answers from both streams: "here are the notable occurrences tagged X" and "here's the aggregate breakdown by X." The two views interoperate without entangling the emission paths.

Caveats¶

Skew between streams: the tail threshold (1s / 10k rows / error) means the notable stream is unrepresentative of average-case workload. Mistake: concluding "most queries are slow" from the notable stream. The aggregate stream is the corrective.
Duplicate work at emission: the source emits both streams concurrently, which costs CPU at the data source. The cost is bounded (one notable message per qualifying query; one aggregate contribution per query) but non-zero.
Schema divergence over time: without discipline, the two streams accumulate incompatible fields. Shared tag model (SQLCommenter) is a useful forcing function — tags apply to both streams in the same form.

Seen in¶

sources/2026-04-21-planetscale-enhanced-tagging-in-postgres-query-insights — Canonical disclosure. Rafer Hazen's post explicitly enumerates the two topics ("The extension publishes to two separate Kafka topics: Individual queries — any query reading more than 10,000 rows, taking longer than 1 second, or resulting in an error. One message is sent per qualifying query… Aggregate summaries — … One message is sent for every query pattern every 15 seconds").