Skip to content

CONCEPT Cited by 2 sources

Observability as code

Observability as code is the posture of treating every observability resource — dashboards, alert rules, SLOs, synthetic checks, recording rules, contact points — as files under version control that are authored, reviewed, diffed, and deployed through the same pipelines as application code. The vendor UI remains for reading and for ad-hoc investigation, but the source of truth is the repository, not the UI.

What falls under the umbrella

The pre-existing wiki entry patterns/alerts-as-code covers alerts specifically. Observability as code is the broader generalisation: the same pull-edit-push discipline applied to every named observability primitive a vendor exposes.

Resource As-code form
Dashboards JSON / YAML / Jsonnet, rendered by Grafana's provisioner
Alert rules YAML / Prometheus-style, backtested before merge (patterns/alerts-as-code)
SLOs Declarative SLO file (target, indicator, window)
Synthetic checks YAML / JSON probe definitions
Recording rules Prometheus recording-rule YAML
Contact points / notifications YAML with tokens / webhooks

Why it matters

  • Review discipline. A dashboard change is a PR; a missing alert is a deleted YAML block in a diff. The sparsely-documented "edit in the UI" anti-pattern is replaced by code-review-grade change control.
  • Portability. As-code resources can be copied across environments (staging → prod) or organisations (ops → ML ops) with variable substitution. UI-hand-authored resources can't.
  • Migration. When the backing observability stack changes (vendor swap, OSS → managed, managed → multi-vendor), as-code resources migrate via tooling; UI-authored resources don't.
  • Agent driveability. As-code resources sit in files the agent already knows how to read and write. This is the point the gcx launch post (sources/2026-04-29-grafana-get-observability-in-the-terminal-for-you-and-your-agents-with-the-gcx-cli-tool) makes: "Pull dashboards, alerts, SLOs, and checks as files. Edit them locally with your agent. Push them back."

The pull-edit-push lifecycle

The canonical loop:

  vendor UI  ←──── push ────── local edit ────── pull ────→  vendor UI
  (source)                    (filesystem)                   (destination)
                             agent or human
                          with full source control
  • Pull. gcx dashboards pull … (or equivalent) fetches the current state from the vendor as local files, ready for editing in any IDE or by any coding agent.
  • Edit. Standard file editing — version control, branch, review, diff.
  • Push. gcx dashboards push … applies the files as the new authoritative state.

A second variant is file-first authoring — the resource is created in the repository from scratch, and the vendor only ever sees the applied result. Either mode keeps the repository authoritative.

Relationship to infrastructure-as-code and GitOps

Observability-as-code is the observability-layer sibling of infrastructure-as-code. The patterns/git-based-config-workflow page captures the more general pattern. Observability-as-code narrows the generalisation to observability resources specifically, where:

  • The resources tend to be many small files (hundreds of dashboards, hundreds of alerts) rather than a single IaC graph.
  • The review burden is shared between ops and product teams rather than centralised on a platform team.
  • Backtesting is essential (for alerts) in a way that infrastructure-as-code doesn't typically have an analogue for.

Seen in

Last updated · 433 distilled / 1,256 read