CONCEPT Cited by 1 source

Ground truth from analyst feedback

Definition

An evaluation methodology where human experts' production decisions (confirm / override) on agent outputs become the ground-truth dataset for measuring and improving agent quality. Unlike synthetic benchmarks, these labels capture real-world judgment on production data.

Mechanism

1. Every agent decision is recorded as a trace (inputs, intermediate steps, output).
2. Human analysts review escalated items in their normal workflow.
3. Each confirm/override label is attached directly to the corresponding trace.
4. The accumulated labeled traces form a benchmark dataset.
5. Any future prompt or agent change is evaluated against this benchmark before deployment.

At Databricks, analyst labels on MLflow traces establish ground truth for the security triage fleet. The dataset captures judgment at scale and enables offline evaluation of prompt changes (Source: sources/2026-07-06-databricks-scaling-security-alert-triage).

Advantages over synthetic evaluation

• Reflects actual production data distribution, not curated examples.
• Captures domain expertise and institutional knowledge implicitly.
• Grows continuously as analysts do their normal work — no separate labeling effort.
• Directly measures the decision boundary that matters in production.

Seen in

• sources/2026-07-06-databricks-scaling-security-alert-triage — analyst labels on MLflow traces as ground truth for Databricks security agents.