Skip to content

SYSTEM Cited by 2 sources

Honeycomb

Honeycomb (honeycomb.io) is a SaaS observability platform built around OpenTelemetry-style distributed traces with a query model optimized for high-cardinality, wide-event data. It's the distributed-tracing backend most cited in the Fly.io corpus on this wiki.

Fly.io's endorsement

From the 2025-03-27 Operationalizing Macaroons post, which includes Thomas Ptacek's explicit retraction of prior skepticism about tracing:

"Most of that is down to OpenTelemetry and Honeycomb. From the moment a request hits our API server through the moment tkdb responds to it, oTel context propagation gives us a single narrative about what's happening. I was a skeptic about oTel. It's really, really expensive. … Once, I was an '80% of the value of tracing, we can get from logs and metrics' person. But I was wrong." (Source: sources/2025-03-27-flyio-operationalizing-macaroons.)

From the 2025-02-12 exit interview with JP Phillips (engineering-side perspective on the same stack):

"Without oTel, it'd be a disaster trying to troubleshoot the system. I'd have ragequit trying. … It was worth the money to pay someone else to manage tracing data." (Source: sources/2025-02-12-flyio-the-exit-interview-jp-phillips.)

Two independent Fly.io voices, two different services (token authority vs. orchestrator), same conclusion: Honeycomb-grade distributed tracing is a load-bearing investment in a platform at Fly.io's scale.

What Fly.io uses it for

  • End-to-end request tracing across micro-services — primary API → tkdb → response. "A single narrative about what's happening."
  • Incident detection — errors in the token system are rare, and when they happen "it's because something has gone so sideways that we'd have to care anyways" — traces make that triage tractable.
  • systems/opentelemetry — the open standard Honeycomb ingests. At Fly.io both sides are in use — OTel in the code, Honeycomb on the receiving end.
  • Prometheus — Fly.io also uses Prometheus metrics ("the standard logging and Prometheus metrics every Fly App gets for free"), but the narrative-tracing part is Honeycomb.

Seen in

Last updated · 200 distilled / 1,178 read