CONCEPT Cited by 1 source

Epistemic vs. aleatoric uncertainty

Definition

Two fundamentally different kinds of uncertainty in ML systems, requiring different responses:

Epistemic uncertainty — uncertainty due to limited knowledge (model limits, judge gaps, insufficient data). Reducible by improving the model/judge, adding data, or refining evaluation criteria.

Aleatoric uncertainty — uncertainty inherent to the task itself (ambiguous inputs, multiple valid answers, subjective tasks). Irreducible regardless of model quality.

In LLM evaluation

In the context of LLM-as-judge evaluation, the distinction is critical because:

• Epistemic uncertainty manifests as judge disagreement that could be fixed (better prompts, domain knowledge, calibration)
• Aleatoric uncertainty manifests as legitimate task ambiguity (multiple valid answers, subjective quality)

Methods that fail to separate them misclassify high-entropy responses as hallucinations (Abbasi Yadkori et al., NeurIPS 2024). Epistemic uncertainty is the more actionable signal, while aleatoric reflects properties no improvement resolves (Ling et al., 2024).

Practical implications

• Don't invest in judge quality for tasks where noise is aleatoric
• Don't accept noise as irreducible when it's actually epistemic
• Require evaluation conclusions to survive judge rotation (epistemic test)
• Stratify results to surface which examples have aleatoric ambiguity

Seen in

• sources/2026-07-14-airbnb-llm-evaluation-infrastructure — Airbnb's dual indeterminacy frame separating these for practical ML iteration