CONCEPT Cited by 1 source

Evaluation noise decomposition

Definition

The practice of decomposing observed score movements in LLM evaluation into their constituent causes before drawing conclusions. A score change can reflect:

1. Model improvement — the actual signal
2. Judge drift — epistemic noise from the evaluator (~1% per run at Airbnb)
3. Reference shift — regenerated references producing different strings (~75% differ across runs)
4. Some combination of the above

Without decomposition, a 2% score improvement is ambiguous — it could be entirely noise. Meaningful differences must survive perturbation: the conclusion holds when you rotate judges, version metrics, and re-stratify samples.

Relationship to statistical testing

Formal significance testing (e.g., t-tests) is a complement, not the full answer. A difference that passes a t-test but flips under judge swap isn't a difference worth shipping on. The question is not only "which average is higher?" but "where do the systems differ, and are those differences meaningful, stable, and product-relevant?"

Seen in

• sources/2026-07-14-airbnb-llm-evaluation-infrastructure — Airbnb's diagnostic framing; Layer 1 addresses noise decomposition before attempting measurement