CONCEPT Cited by 1 source

Dual indeterminacy¶

Definition¶

Dual indeterminacy is the recognition that noise in LLM evaluation has two independent sources requiring separate diagnosis and remediation:

Epistemic uncertainty — model or judge limitations that can be reduced with better judges, more data, or improved prompts. Actions: improve judge quality, increase sample size, refine evaluation criteria.
Aleatoric uncertainty — inherent task ambiguity that no judge improvement resolves. The task itself admits multiple valid answers. Actions: acknowledge the ceiling, stratify analysis, adjust expectations.

When the real signal from a model improvement is 1–3%, and judge drift alone accounts for ~1% per run, conflating these two sources produces wrong conclusions about whether a change is actually an improvement.

Why it matters¶

Without separating these sources, teams fall into two traps: - Over-investing in judge quality for tasks where the noise is aleatoric (unfixable) - Under-investing in judge quality for tasks where drift is epistemic (fixable)

The frame was formalized by Abbasi Yadkori et al. (NeurIPS 2024), showing that methods failing to separate epistemic from aleatoric uncertainty misclassify high-entropy responses as hallucinations.

Seen in¶

sources/2026-07-14-airbnb-llm-evaluation-infrastructure — Airbnb's Layer 1 diagnostic framing; ~75% of references differ across runs, ~1% judge drift per run