Skip to content

PATTERN Cited by 1 source

Agent-assisted label validation

Use the same agent you are trying to evaluate to assist with validating its evaluation labels. When the agent's quality clears an alignment bar against human judges, it can aggregate signals, resolve ambiguous customer feedback, derive causal chains, and propose a ground-truth RCA + world-snapshot query set — shifting humans from assembling labels out of raw signals to validating and refining agent output.

Intent

Product-feedback-driven label creation floods the label pipeline with candidate material. Manually turning each candidate into a full evaluation label (precise RCA + reconstructed world snapshot + noise-expanded signal set) requires human reviewers who scale linearly while label volume scales with adoption. Manual review cannot keep up; labels are dropped, diversity suffers, and the eval platform becomes the agent's bottleneck.

Mechanism

  1. Establish alignment. Run a controlled alignment study: have both the agent and human judges produce RCAs on the same set of candidate cases. Measure agreement on thoroughness / specificity / accuracy. Only when alignment clears a quality bar does the agent get promoted to the validation path.
  2. Agent processes feedback + investigation telemetry. The agent aggregates related signals, derives causal relationships, and resolves ambiguous references in user feedback (e.g. "it was slow" → "elevated latency in service X"). Unlike a production investigation, the agent is told the ground truth at this stage and asked to construct the causal chain that would connect the problem statement to the root cause.
  3. Score the agent's output. Confidence scores across thoroughness / specificity / accuracy. Below threshold → human review. Above threshold → lighter-touch human validation (refining agent output, not assembling from scratch).
  4. Humans validate, don't assemble. The human's job becomes checking and editing the agent's proposed RCA + snapshot queries. Much faster than starting from raw signals.
  5. Continuous recalibration. Alignment studies re-run periodically as the agent, the model, or the labelling problem drifts.

Why it works

  • Breaks the linear-scaling ceiling on label validation. Agent assistance is a judge-style application used up the funnel from scoring — the same capability that makes judges viable makes this step viable.
  • Precision-appropriate. RCA construction is high-precision and low-margin-of-error ("just like diagnosing the root cause of an issue"). Apply only when agent quality has clearly reached the level where this is possible; hard-gate on the alignment study.
  • Self-improving loop. Better agent → better label candidates → better evaluation → better agent. The platform's throughput rises with every agent-quality step.

Tradeoffs

  • Circularity risk. Using agent A to help validate labels that will evaluate agent A creates feedback structure. Mitigate by: having the validation step produce auditable artifacts (the agent's reasoning is recorded); keeping human spot-checks on a fraction of validated labels; recalibrating alignment periodically; never letting the agent under test mark its own failure scenarios as passing.
  • Gated by quality threshold. This pattern doesn't work until the agent is already good enough — it's an amplifier of existing quality, not a bootstrapping mechanism. For a new agent, start with manual labelling.
  • Confidence-score calibration is critical. If the threshold is too lax, low-quality labels leak into the set; too strict, and the throughput win evaporates. Track drift.
  • Not a safety proof. Agent-assisted validation doesn't prove the agent is safe — it proves it can construct RCAs from known outcomes, which is weaker. Mutating production actions still need separate guardrails.

Reported impact

Datadog: validation time per label dropped >95% in a single week after agent-assisted validation came online (Source: sources/2026-04-07-datadog-bits-ai-sre-eval-platform). Label quality, measured as fraction of RCAs that would survive a "5 Whys" postmortem, rose ~30%. Higher-quality labels in turn enabled concepts/trajectory-evaluation in the downstream evaluation — the compounding benefit.

Seen in

Last updated · 200 distilled / 1,178 read