Skip to content

PATTERN Cited by 1 source

Continuous agent evaluation framework

Pattern

Run a continuous evaluation loop over agent traces that tests not just the agent but the entire context stack it relies on — tools, context documents, system prompts, and domain assumptions — using multiple scorer types before any change shapes production behavior.

Structure

  1. Standardised traces — MLflow tracing across all agent adapters; each interaction is a structured execution path (model calls → tool calls → intermediate steps → final answers)
  2. Multi-scorer evaluation — run against traces:
  3. Deterministic correctness scorers
  4. SQL-grounding scorers (did the agent query the right tables with correct logic?)
  5. Cache-aware cost scorers (is the agent efficient?)
  6. LLM-judge scorers with domain-expert alignment
  7. Context-stack testing — markdown can drift from pipelines and table definitions; system prompts evolve; tool contracts change; domain assumptions age. The framework tests all of these together.
  8. Observable substrate — AI Gateway routes model traffic → inference tables land in Unity Catalog → analysable with SQL

Why test the context, not just the agent

"The context layer itself needs evaluation: markdown can drift from pipelines and table definitions, system prompts evolve, tool contracts change, and domain assumptions age. Traces and evals let us test the agent, the tools, and the context it relies on together before any of them shape production behavior." (Source: sources/2026-07-21-databricks-why-rd-data-belongs-in-the-lakehouse-and-why-agents-need-it)

Seen in

Last updated · 590 distilled / 1,788 read