Skip to content

GRAFANA 2026-10-19 Tier 2

Read original ↗

Six improvements that make Grafana Assistant better at helping your workflows

Summary

A Grafana Labs roundup post (2026-10-19) that ties together six months of engineering work on Grafana Assistant — the observability-integrated AI agent inside Grafana Cloud — into one thesis: "others build agent loops, we build evaluation loops." It is a digest with links to seven deeper Unprompted (Grafana engineering blog) write-ups, but it carries enough concrete architecture and operational numbers to clear the borderline-include bar. The load-bearing idea is a closed feedback cycle across three surfaces that mutually reinforce: better evaluations reveal where an agent struggles → findings guide changes to its tools and context → production telemetry (via Agent Observability) confirms whether the change worked and where to look next. The six improvement areas: dashboarding micro-optimization, an evaluation loop that scores agent work not just final text, the agent-ergonomic gcx CLI, progressive tool discovery to cut context overhead, a redesigned retrieval system, and arena-style evaluation of Investigations against real incidents.

Key takeaways

  1. Dashboarding is 27% of Assistant conversations (by mid-2026) and was the first target of a measurement-driven optimization pass. Establishing a baseline with Agent Observability + reproducible tests, then iterating over ~10 weeks, moved three production metrics: error rate 2.47% → 0.65%, p95 latency per operation 63s → 28.8s, cost per request −18%. The test catalog grew from 78 scenarios to nearly 200. (Source — Dominik Broj's dashboarding write-up.) This is a textbook patterns/measurement-driven-micro-optimization loop whose target metric is agent error/latency/cost rather than CPU.

  2. "Evaluating the final answer alone misses much of what determines whether the task succeeded." An agent can produce a convincing response while the underlying work is wrong — a panel renders successfully but shows the wrong aggregation; an investigation names a plausible cause without enough evidence. The evaluation loop therefore separates the behavior you want from the prompts and code that implement it: scenarios define realistic tasks + success criteria; graders examine queries, tool calls, saved state, and conclusions (not just the text); repeated runs expose inconsistent behavior; comparisons track quality alongside cost and latency. (Source — Yasir Ekinci's evaluation-loop write-up.) This is patterns/snapshot-replay-agent-evaluation with trajectory-level grading via concepts/llm-as-judge.

  3. Coding agents are inside the eval loop. "Coding agents also help inspect failures and propose changes, which go back through the evaluation loop for verification and human review." — a propose→verify→human-review cycle where the regression harness is the verification gate.

  4. gcx is a CLI designed by watching agents fail. Watching agents use the CLI revealed repeated failure modes: guessing flags, retrying unsuccessful commands, and requesting far more output than they needed. The fixes made the interface informative: commands describe their capabilities, errors explain how to recover, and agents can inspect available fields and filter results before bringing them into context. "Knowledge that once required a separate playbook increasingly lives in the tool itself, where it can stay aligned with the implementation." Core design question: what should the interface tell the agent so it can take the next step correctly? (Source — Ward Bekker's gcx write-up.) See systems/gcx-cli and concepts/agent-ergonomic-cli.

  5. Progressive tool discovery cut initial tool-definition overhead by 99.1%. As the number of capabilities grows, loading every tool schema upfront is unaffordable. Assistant instead "starts with compact entry points and retrieves the relevant manual" when needed. Across the full registered tool surface, initial tool definitions shrank from roughly 54,500 estimated tokens to 504 — a 99.1% reduction in that specific context overhead. The post is careful: "actual session savings depend on the capabilities available and used" and "smaller prompts are only helpful if discovery does not introduce excessive steps or delays" — the complete workflow is still measured. (Source — Cyril Tovena's progressive-tool-discovery write-up.) Canonical datum for concepts/progressive-capability-disclosure and concepts/token-overhead.

  6. Retrieval quality is a representation-and-permission problem, not a "put documents in an index" problem. Ioanna Armouti's search write-up records four decisions: (a) for dashboards, embedding raw panel JSON drew attention toward rendering details — representing a panel through its folder, dashboard, row, description, and queries made its purpose clearer; (b) incident summaries needed different treatment from noisy activity feeds; (c) permissions must be checked at retrieval time, not index time, because access can change after indexing; (d) retrieval, tool discovery, and dashboard editing are the same problem — making the right information reachable while controlling how much enters the agent's context. See concepts/retrieval-ranking-funnel / concepts/hybrid-search.

  7. Investigations were evaluated arena-style against real resolved incidents — a concept "borrowed from the world of competitive programming" (comparing agent variants head-to-head). Synthetic telemetry had been insufficient: real incidents bring incomplete data, competing explanations, and distracting signals. Two concrete fixes surfaced: one agent inferred too much from an alert's name (fetching the actual alert definition gave a better starting point); another avoided a useful tool because its instructions described the wrong query language (filtering retrieved background reduced irrelevant context). Scores on the internal difficult-incident suite moved 52% → 69% during a week of iteration. (Source — William Dumont's Investigations write-up.)

  8. Production evaluator rebuild + a typed-decision grader experiment ("Jev"). As agents did longer work sequences, the original production evaluator "increasingly missed failures because it focused too heavily on response text." The rebuild: show the evaluator more of the agent's work, let it recognize partial completion, and test its judgments against a stronger reference reviewer. They then tested Jev, "a model designed for structured decisions" — substantially faster and cheaper, but it agreed with the reference reviewer much less well than the strongest generative judge. So they kept the stronger judge for the main quality score and introduced Jev as an optional provider with a trace-triage template — "its suitability for that different task still requires representative validation." (Source — Sven Großmann & Yasir Ekinci's "Judge Jev" write-up.) A cost/quality-tiered judge: cheap typed-model for triage, expensive generative judge retained for the score — a cheap-vs-expensive split in the grading layer, decided by an evaluate-then-keep-what-works pass.

  9. Agent Observability is the dogfooding ↔ product bridge. "Building and operating Assistant gives us difficult, practical questions about agent behavior. Agent Observability helps us answer them through traces, evaluations, and comparisons — and makes those capabilities available to customers building agents of their own." The internal eval infrastructure is productized as a Grafana Cloud offering.

Systems / concepts / patterns extracted

Operational numbers

Metric Before After Note
Dashboarding share of Assistant conversations — 27% by mid-2026
Dashboarding error rate 2.47% 0.65% ~10 weeks
p95 latency per operation (dashboarding) 63 s 28.8 s per individual operation
Cost per request (dashboarding) — −18% over the same window
Dashboarding test catalog size 78 scenarios ~200 scenarios grew during the pass
Initial tool-definition tokens (full registered surface) ~54,500 504 −99.1% context overhead
Investigations difficult-incident-suite score 52% 69% one week of iteration
Blinded judge sample (Jev vs reference) — — agreement "much less" than strongest generative judge (no number)

Caveats

  • Digest, not a deep-dive. This post is a roundup; the architecture lives in seven linked Unprompted / Medium posts (dashboarding, evaluation loop, gcx, progressive tool discovery, search, Investigations arena, Judge Jev) — none of those are ingested here. The numbers are reported from those posts, not re-derived.
  • The 99.1% is a scoped claim. It is the reduction in initial tool-definition tokens, explicitly not end-to-end session savings — the post flags that discovery can add steps/latency and that the complete workflow is still what's measured.
  • Scores are benchmark figures, not production SLAs. The 52%→69% Investigations number "measures performance on those benchmark cases"; the dashboarding figures are production metrics but over a specific ~10-week window.
  • Jev not validated for its new task. Grafana explicitly kept the stronger generative judge for the main quality score; Jev is admitted only as an optional trace-triage provider pending "representative validation." No Jev architecture, size, or agreement number is disclosed.
  • No LLM / vector-DB / embedding-model disclosure for Assistant, consistent with prior Grafana Assistant posts.
  • ObservabilityCon 2026 framing. The post closes on an event CTA; the architecture content (not the event promo) is what carries it over the scope bar.

Source

Last updated · 854 distilled / 2,448 read