Skip to content

Grafana Labs

Grafana Labs Engineering blog (grafana.com/blog). Tier-2 source on the sysdesign-wiki. Grafana Labs is the commercial steward of the Grafana open-source observability stack — dashboards, Loki (logs), Prometheus-compatible metrics backends, Tempo (traces), Pyroscope (profiling), plus managed offerings (Grafana Cloud Logs, Grafana Cloud Metrics, Grafana Cloud Traces).

Editorial character

The blog carries a mix of deep engineering posts, architectural announcements (acquisitions, major releases), and product-launch content. GrafanaCON announcements tend to be higher PR load but often disclose genuine architectural motivation for new features; those are ingested when the architecture content exceeds the ~20 % threshold per AGENTS.md scope rules.

Key systems

  • systems/grafana — the flagship dashboarding / visualisation platform; the canonical "single pane of glass" for metrics / logs / traces / profiles across many backends (Prometheus, Loki, Tempo, Pyroscope, and third-party data sources). Ubiquitous across other companies' observability stacks on this wiki.
  • systems/loki — Grafana Labs' horizontally-scalable, HA, open-source log-aggregation system. Label-based indexing over object-storage chunks. Designed as the Prometheus-analogue for logs: same data model, same operational shape, but for log lines.
  • systems/tempo — Grafana Labs' horizontally-scalable, object-storage-backed distributed-tracing backend. Stores trace data as columnar vParquet blocks, queries with TraceQL (incl. TraceQL metrics for RED/service-graphs derived from spans), and ingests via Kafka (3.0+). The traces leg of the stack alongside Loki, Mimir, and Pyroscope. See the 2026-10-01 Tempo 3.1 release for its Kafka-client hardening, sampling-aware metrics, trace redaction, and span pruning.
  • systems/grafana-mimir — Grafana Labs' horizontally-scalable Prometheus-compatible metrics database. Forked from Cortex and rearchitected to eliminate write-path replication and make object storage the single source of truth — the architectural template for Pyroscope 2.0.
  • systems/pyroscope-2 — 2026-04-22 launch. Ground-up rearchitecture of the original Pyroscope continuous-profiling database. Native OpenTelemetry Profiles (OTLP) ingest; applies the same Cortex-to-object-store move as Mimir, adapted for profiling data's large payloads, heavy symbolic information, and bursty query patterns.
  • systems/pyroscope — Pyroscope 1.x, the Cortex-based predecessor; retained to document the lineage and the architectural motivation for 2.0.
  • systems/cortex — the CNCF horizontally-scalable Prometheus-compatible metrics project that is the common architectural ancestor for Mimir, Loki, and Pyroscope 1.x.
  • systems/logline — 2026-04 acquisition (founder Jason Nochlin, ex-Teleport-Data CEO). Secondary indexing technique specifically for high-cardinality attributes over object storage — designed to close Loki's structural weakness on needle-in-the-haystack queries (UUID / request-ID lookups). Reported benchmark: 3.5 TB → 8 GB scanned for a UUID query (99.7 % reduction in data scanned).
  • systems/grafana-cloud-logs — Grafana Labs' managed Loki offering; first-launch surface for Loki features under managed operational control before Loki OSS release.
  • systems/grafana-faro — open-source frontend observability web SDK (github.com/grafana/faro-web-sdk) shipping browser errors / performance / logs to Grafana-compatible backends. Powers Frontend Observability (RUM).
  • systems/grafana-synthetic-monitoring — Grafana Cloud's managed synthetic monitoring: scheduled, scripted/declared checks from known probe locations. The outside-in detection signal; pairs with Frontend Observability via a direct cross-signal link.
  • systems/grafana-frontend-observability — Grafana Cloud's RUM product (Faro-powered). The inside-out impact signal — Core Web Vitals, JS errors, and Session Replay from real sessions — that supplies the blast radius synthetic checks can't see.
  • systems/grafana-cloud — Grafana Labs' fully-managed observability platform. Unifies the Loki / Mimir / Pyroscope / Tempo / Faro / synthetics surface under one vendor-operated control plane; the target of gcx and the "deep link into Grafana Cloud the moment a human needs to look" handoff.
  • systems/gcx-cli — 2026-04-29 open-source launch (github.com/grafana/gcx). Grafana Cloud's agent-ergonomic CLI covering the full observability lifecycle: instrumentation (OpenTelemetry wiring + flow validation), alerting / SLOs / synthetics, frontend + application + K8s observability onboarding, and everything-as-code (dashboards + alerts + SLOs + checks as pull-edit-push files). Ships seven explicit agent-primary commitments — stable JSON/YAML with version-stable field names, documented + consistent exit codes, machine-readable command catalog with destructive-command tagging, auto-detection of Claude Code / Cursor harnesses with GCX_AGENT_MODE=true override, destructive-operation confirmation, kubectl-style named contexts for multi-stack sessions, and deep links into the Grafana UI. Also ships a bundled portable agent-skills library (observability setup, alert investigation, SLO management, synthetic-check investigation) usable in any .agents- skill-convention harness.
  • systems/grafana-assistant — 2026-05-01 canonical wiki ingest. Grafana's AI assistant surface embedded inside Grafana Cloud with, from GrafanaCON 2026, self-managed availability. The 2026-05-01 post discloses the infrastructure memory subsystem: a swarm of AI agents walks the stack's Prometheus / Loki / Tempo data sources, extracts a five-category schema (identity + metrics + topology + dependencies + log structure) per service group, stores it in a vector database, and retrieves it via semantic search "in milliseconds". Refreshes weekly with manual-trigger escape hatch. ACL-propagated via data-source-source-of-truth linkage (memories inherit the access-control boundary of the data sources they were generated from — no ACL copy, ACL enforced at query time). Zero-configuration commitment — "If you have metrics, you get this infrastructure memory capability." First wiki instance of concepts/agent-memory at the observability-stack substrate (distinct from Meta's precompute-engine code-substrate and Cloudflare's agents-that-remember conversation-substrate).

Recent articles

  • 2026-10-01 — Tempo 3.1: Kafka ingestion hardening, sampling-aware TraceQL metrics, query-driven trace redaction, span pruning — a release post for Grafana Tempo carrying real distributed-systems content. Kafka ingestion gains TLS/mTLS + 4 SASL mechanisms, KIP-392 rack-aware fetching (client_rack) to cut cross-AZ transfer cost (concepts/egress-cost), and configurable producer_compression (concepts/compression-codec-tradeoff) so gzip-only Kafka-protocol backends like Azure Event Hubs work. TraceQL metrics gains sampling-aware extrapolation (reads the OTel tracestate sampling rate — see systems/opentelemetry), inter-query arithmetic, and a ~2× span-only fetch read path. Also: query-driven trace redaction rewriting immutable object-storage blocks (pausing per-tenant compaction, with a mixed-version over-delete hazard), read-time span pruning, and a zero-allocation pooled-buffer metrics-generator hot path (concepts/metric-cardinality).

  • 2026-08-24 — From failed check to real user impact: Pairing Synthetic Monitoring and Frontend Observability in Grafana Cloud — a Grafana Cloud post arguing that Synthetic Monitoring (active, outside-in) and Frontend Observability (RUM, inside-out, Faro-powered) are complementary, not redundant, and together form closed-loop reliability. Load-bearing epistemics: a synthetic check is a controlled experiment (fixed script/target/probe/runtime, so deviations are attributable to the system, not the test — trustworthy enough to alert on) but is structurally blind to who / how many / how badly (blast radius is a real-traffic property); RUM is a population sample (every session unique; signal is in the aggregate pattern). The reactive loop pivots from a failed check into real sessions via a direct link built "to skip manual timestamp-matching and url-hunting" (synthetic-alert-to-rum-pivot) to get scope + duration + severity; the proactive loop turns RUM-surfaced gaps into new checks so the suite "evolves with real traffic instead of aging against it" (rum-gap-to-synthetic-check). Worked example: /checkout browser check fails from Frankfurt at 09:13; RUM shows the real-world start at 09:07; output is "the checkout flow has degraded for ~8% of EU users since 09:07" replacing "a check failed, investigating." Creates 3 systems (systems/grafana-synthetic-monitoring, systems/grafana-frontend-observability, systems/grafana-session-replay), 4 concepts (synthetic-monitoring, concepts/real-user-monitoring, closed-loop-reliability, cross-signal-correlation-for-triage), and 3 patterns (synthetic-alert-to-rum-pivot, rum-gap-to-synthetic-check, cross-signal-correlation-for-triage). Caveat: product post — the synthetic-vs-RUM epistemics and closed-loop framing are reusable, but the workflow (Page button, direct link) is Grafana-Cloud-specific; the 8%/09:07 figures are illustrative, not disclosed production metrics.

  • 2026-06-24 — Post-incident review for TanStack npm supply chain ransom incident — Detailed post-incident review of the May 2026 TanStack "Mini Shai-Hulud" supply chain attack. Self-hosted runner credential leak → full repo exfiltration → ransom demand (not paid). Primary architectural remediations: token broker for short-lived scoped credentials, compartmentalized GitHub organizations, DockerHub → Google Cloud Artifact Registry migration. Operational scale: 1,500 security PRs, 280 GitHub app audits, 1,200 repos scanned, 8-day global code freeze gated on token-broker adoption. Independent Mandiant audit corroborated no customer/production impact. Key lesson: credential inventorying is harder than rotation mechanics (one missed credential enabled the entire exfiltration 5 days post-initial-rotation).

  • 2026-05-01 — Faster fixes, less context sharing: how Grafana Assistant learns your infrastructure before you even ask — launch post for the infrastructure memory subsystem of Grafana Assistant. Canonical first wiki ingest for Grafana Assistant itself (earlier 2026-04-22 AI-Observability and Assistant-everywhere launch posts were skipped as pure marketing; this one passes the borderline-case architecture bar). Load-bearing disclosures: (1) a four-stage swarm of AI agents discovery pipeline — data source discovery → parallel Prometheus metrics scans → Loki/Tempo enrichment → structured knowledge generation; (2) a fixed five-category service-knowledge schema per discovered service group (identity + purpose, key metrics + labels, deployment topology, dependencies, log structure); (3) vector-database storage with semantic-search retrieval in milliseconds — the memory tier is an embedding-keyed chunk store, not a structured catalogue or a raw-log index; (4) weekly automatic refresh + manual-trigger escape hatch discipline; (5) link-don't-copy ACL propagation — each memory is tagged with its originating data source and the data source's ACL is enforced at retrieval time (not copied onto the memory); (6) the telemetry-as- discovery-substrate architectural bet — "Your existing telemetry data is the input … If you have metrics, you get this infrastructure memory capability" — which is what makes the zero-configuration UX commitment possible. Creates 1 system (systems/grafana-assistant), 4 concepts (concepts/agent-memory, telemetry-as-discovery-substrate, concepts/agent-memory, concepts/agent-memory, service-group), and 3 patterns (patterns/specialized-agent-decomposition, five-category-service-knowledge-schema, weekly-refresh-cadence-for-agent-context). Complementary to the 2026-04-29 gcx launch: where gcx is the terminal-hosted agent surface on Grafana Cloud, the Assistant is the UI-embedded agent surface; both draw from the same underlying observability-stack control plane. Caveats: no disclosure of underlying LLM, vector DB, or embedding model; no operational numbers beyond "milliseconds" retrieval; service-group detection heuristic not disclosed; weekly-refresh partial-failure handling undisclosed; "swarm of AI agents" framing is marketing to an unspecified extent (no agent inventory or inter-agent protocol disclosed).

  • 2026-04-29 — Get observability in the terminal, for you and your agents, with the gcx CLI tool — launch post for gcx, Grafana Cloud's open-source CLI built agent-primary. Canonicalises (1) Grafana as the first observability vendor to ship the full agent-ergonomic CLI commitment list as a single open-source tool, (2) observability-as-code as a first-class pattern beyond alerts-as-code — dashboards + alerts + SLOs + synthetics all pull-edit-push as files, (3) five canonical agent-conversation shapes that the CLI enables ("Why did this endpoint get slower this week?" → traces + latency histograms; "Is my new query efficient?" → PromQL against real backend + iterate; "Are we meeting the SLO for checkout?" → burn-rate-before-writing; "This alert is noisy, fix it." → rule + history + proposed threshold), (4) deep-link handoff as the human-in-the-loop primitive — agent does the work in the terminal, emits a precise URL when human review is needed, (5) the direct-exec-over-wrapper design stance: "The agent calls gcx the way it already calls git or kubectl. No wrapper, no shim, no bespoke integration layer", (6) a bundled portable agent-skills library covering observability setup, alert investigation, SLO management, and synthetic-check investigation, portable across any .agents-skill-convention harness including Claude Code. Creates 3 systems (systems/gcx-cli, systems/grafana-cloud, +Faro extension), 5 concepts (concepts/observability, concepts/structured-output-reliability, exit-code-semantics, concepts/machine-readable-documentation, named-contexts-for-multi-stack), and 3 patterns (patterns/observability, auto-detect-agent-context, destructive-operation-confirmation-as-agent-guardrail, deep-link-to-ui-from-cli). No quantitative benchmarks disclosed; exact command shapes deferred to the github.com/grafana/gcx repository.

  • 2026-04-22 — Introducing Pyroscope 2.0: faster, more cost-effective continuous profiling at scale — GrafanaCON 2026 launch announcement for the ground-up rearchitecture of Grafana Labs' open-source continuous-profiling database Pyroscope 2.0. Retires the Cortex-based architecture of Pyroscope 1.x and applies the same principles Mimir adopted — eliminate write-path replication, decouple reads from writes, make object storage the single source of truth — adapted for profiling data's large payloads, heavy symbolic information, and bursty query patterns. Ships native OpenTelemetry Profiles (OTLP) ingest (the Profiles signal recently reached alpha). Launch-post caveat: quantitative benchmarks not published in this post — directional credibility via Mimir precedent.
  • 2026-04-22 — Grafana Labs acquires Logline to accelerate needle-in-the-haystack log queries — acquisition announcement at GrafanaCON 2026. Logline brings high-cardinality-over-object-storage indexing to Loki, closing its structural weakness on UUID / request-ID lookups. Headline: 3.5 TB → 8 GB scanned (99.7 % reduction) on the missing-needle worst case. Launched in Grafana Cloud Logs limited private preview; Loki OSS release targeted for the next major version "in the coming year." Canonical wiki articulation of Loki's label-based indexing thesis and the

secondary-index-for-high-cardinality-over-object-storage pattern.

Architectural themes surfaced

  • Cortex → object-store rearchitecture across all three observability signals. Grafana Labs' metrics (Mimir), logs (Loki), and continuous profiles (Pyroscope 2.0) databases all started from the same Cortex foundation and are converging on the same rearchitecture: eliminate write-path replication, decouple reads from writes, make object storage the single source of truth. The per-signal engineering is the adaptation to that signal's data shape. See observability-db-rearchitecture-cortex-to-object-store for the unified framing.
  • Label-based indexing over full-text indexing for logs. Loki's design is deliberately the Prometheus-style opposite of Elasticsearch: index labels only (service, cluster, environment), store content on cheap object storage. This keeps storage cost low — at the price of needle-in-haystack queries being slow until a secondary index lands.
  • Secondary indexes as complement, not replacement. Rather than regress to full-text-indexed storage costs to handle UUIDs, layer a narrow secondary index specifically for high-cardinality lookups (Logline). Preserves the base design's cost envelope.
  • Managed-first rollout for new architecture. Major features (e.g., Logline-powered needle-in-haystack queries) launch in Grafana Cloud Logs under Grafana Labs operational control before OSS general availability — a pattern shared with other infra vendors that also ship OSS + managed.
  • Continuous profiling as first-class observability signal. With OpenTelemetry's Profiles signal reaching alpha and Pyroscope 2.0 shipping native OTLP profiling ingest, profiling joins metrics/logs/traces as a standard signal with a standard wire format. See concepts/observability and default-continuous-profiling.
  • Agent-primary CLI surface for the managed-platform control plane. The 2026-04-29 gcx launch positions Grafana Cloud's full observability lifecycle (instrumentation / alerts / SLOs / synthetics / frontend / app / K8s / as-code) behind a single CLI designed for AI agents as the primary caller. The seven shipping commitments — stable JSON/YAML with version-stable field names, documented exit codes, machine-readable command catalog, auto-detected agent-mode, destructive-op confirmation, named contexts, deep-link UI handoff — together constitute Grafana's bet on "agentic coding tools belong in the terminal. CLIs match how models actually reason — text in, text out, stable exit codes — and they compose with every credential and config the developer already has." The explicit rejection of wrapper / shim layers ("The agent calls gcx the way it already calls git or kubectl. No wrapper, no shim, no bespoke integration layer.") makes gcx a peer instance of the Cloudflare (cf) and Fly.io (flyctl) agent-ergonomic-CLI lineage at the observability altitude. See concepts/agent-ergonomic-cli.
  • Observability-as-code generalises alerts-as-code. The gcx post lifts the pull-edit-push lifecycle from alerts specifically (Airbnb's canonical) to every observability resource — dashboards, alerts, SLOs, synthetic checks, recording rules, contact points. Grafana Cloud becomes the first vendor-managed instance of observability-as-code as a named shipping commitment rather than an OSS convention.

Source character (for future ingests)

  • Deep posts on Loki / Prometheus / Tempo / Pyroscope internals are ingest candidates.
  • GrafanaCON announcements are borderline — include if the architecture motivation is real (e.g., Logline). Skip if the body is event roundup or award show recap (e.g., GrafanaCON 2026 announcements roundup, Golden Grot awards).
  • Product-PR posts on managed-cloud pricing, feature-parity tables, or integration-count claims are out of scope.
Last updated · 766 distilled / 2,225 read