Skip to content

CONCEPT Cited by 1 source

Single-sink telemetry architecture

Single-sink telemetry architecture is the structural posture of routing observability traffic (traces / logs / metrics) directly from instrumented clients to a single durable analytical store through one managed ingestion endpoint — collapsing away the conventional intermediate-broker hop (Kafka / Pulsar / Kinesis) that traditional multi-hop telemetry pipelines use.

What it replaces

Conventional telemetry pipeline:

clients ──► OTel collector ──► message broker (Kafka/Pulsar) ──► consumer service ──► storage

Single-sink shape:

clients ──► managed OTel receiver ──► storage (lakehouse / TSDB / object store)

The defining property: one hop, one schema boundary, one operational system to own.

Canonical instance: Databricks Zerobus + UC OTel Trace Tables (2026-05-22)

The 2026-05-22 OTel-tracing launch frames the architecture explicitly:

"With a 'single-sink' architecture, Zerobus Ingest simplifies observability by streaming data directly to the lakehouse. Existing OLTP-compatible collectors can point directly to this endpoint via gRPC, entirely bypassing intermediate message buses like Kafka." — Source: sources/2026-05-22-databricks-observability-any-agent-anywhere-otel-unity-catalog

Zerobus Ingest is the managed receiver; UC OTel Trace Tables are the storage destination; the single-sink claim is the absence of a broker between them.

Structural arguments for single-sink

Argument Detail
Fewer hops Each hop is a failure-mode / latency / throughput-bottleneck candidate. Removing one collapses the surface area.
One schema boundary Multi-hop pipelines often re-serialize at each hop (OTel proto → Avro → Parquet). Single-sink lets the receiver write the storage format directly.
Operational simplification No broker to size, scale, secure, patch, monitor, upgrade. Managed ingestion absorbs that.
Lower latency to query Trace ingest → queryable storage in one path; broker buffering removed.
Drop-in compatibility OTLP-compatible collectors that already exist can re-point at the new endpoint without re-architecture.

Structural arguments against (when single-sink is the wrong shape)

  • Multi-destination fan-out. If telemetry must land in two places (lakehouse for analytics + APM for real-time alerting + cold archive for compliance), a single sink can't satisfy all destinations. Brokers fan out cleanly; single-sink doesn't.
  • Buffering for back-pressure. Brokers absorb bursts that overwhelm downstream consumers; single-sink pushes back-pressure to the client (OTel SDK retries, client-side buffer, or drops).
  • Cross-team aggregation. Brokers are natural rendezvous points where multiple producers and multiple consumers meet; collapsing the broker collapses that property.
  • Replay. Brokers retain events for replay; a write-direct-to-storage path requires the storage layer itself to support replay (Delta CDF / Iceberg time travel can — but the property has to be designed in, not assumed).

When the single-sink trade is favourable

The 2026-05-22 architecture argues this for agent traces specifically:

  • One destination is the answer — UC Delta is the lakehouse the customer already governs; alternative destinations are not part of the design intent.
  • Burst tolerance comes from managed receiver — Zerobus is "fully managed, serverless" with the implication that the platform absorbs spikes, though the post does not benchmark burst behaviour.
  • Replay is provided by storage — Delta + CDF gives the same replay property a Kafka topic would, on the storage side.
  • Cross-team aggregation lives in UC — the lakehouse is itself the aggregation point; teams query the same tables.

The trade favours single-sink when the lakehouse is already the canonical analytical destination and when the workload is not real-time alerting (which still wants APM / Prometheus / sub-second tooling alongside).

Relationship to other concepts

Caveats

  • The marketing argument is "fewer hops". The structural argument is "fewer systems to operate". These are correlated but not identical — a single highly-complex managed receiver can be operationally heavier than two simple components, depending on where the operational burden actually lands.
  • "Bypassing Kafka" is not always a win. Kafka exists in many telemetry pipelines for legitimate reasons (multi-consumer fan-out, durable replay, cross-team rendezvous). Single-sink architecture works when those reasons don't apply for the specific workload.
  • The claim is rarely benchmarked. Vendor framings (Zerobus, similar managed-ingest products) emphasise architectural elegance over throughput / latency comparisons.

Seen in

Last updated · 542 distilled / 1,571 read