PATTERN Cited by 1 source

Analyst-labeled trace as ground truth¶

Pattern¶

Record every agent decision as a full trace (input, intermediate steps, output). When a human expert reviews the decision in their normal workflow, attach their confirm/override label to the trace. Accumulate labeled traces into a benchmark dataset that evaluates any future agent or prompt change before deployment.

When to use¶

Agent systems where outputs are reviewed by domain experts as part of normal operations.
When synthetic benchmarks don't capture the real decision boundary.
When you need a continuously growing evaluation set without a separate labeling effort.

Structure¶

Agent decision → MLflow trace (full audit) → Analyst reviews
                                                    │
                                              label attached
                                                    │
                                                    ▼
                                          Ground truth dataset
                                                    │
                                                    ▼
                                    Offline evaluation of prompt changes

Key insight¶

Analyst labels captured during normal workflow create an ever-growing evaluation benchmark without dedicated labeling sprints. At Databricks, this approach enabled rigorous offline testing of prompt changes against production-grade ground truth (Source: sources/2026-07-06-databricks-scaling-security-alert-triage).

Seen in¶

sources/2026-07-06-databricks-scaling-security-alert-triage — MLflow traces labeled by IR analysts as ground truth for Databricks security triage agents.