PATTERN Cited by 1 source
Deterministic evaluation cache¶
Intent¶
Make LLM evaluation reproducible, efficient, and comparable across runs by caching at the per-sample level, eliminating noise from judge drift and reference regeneration.
Mechanism¶
Cache on two independent axes:
- Reference axis — key:
(sample_identifier, reference_generation_config)→ cached reference output - Judge axis — key:
(sample, model_output, judge_config, metric)→ cached score
When inputs are identical to a prior evaluation, cached results are returned directly. Only genuinely different inputs trigger new judge/reference computation.
Properties¶
- Deterministic: identical inputs always produce identical evaluation results
- Durable partial progress: a failed job at example N resumes from cache
- Incremental: new candidates/metrics reuse existing cached outputs
- Efficient: >50% of model outputs across candidates are identical strings (Airbnb measurement)
Why this beats probabilistic approaches¶
| Approach | Problem |
|---|---|
| Majority voting | Converges to judge's central tendency, not accuracy |
| Bayesian noise modeling | Same infra cost as cache, without reproducibility |
| Repeated sampling | Resource-prohibitive at scale (examples × systems × judges) |
| Per-sample cache | Eliminates noise at source; simpler, cheaper |
Prerequisite for¶
- patterns/micro-adapter-hotfix — same-day validation requires deterministic measurement
- patterns/end-to-end-seam-validation — combined-path measurement inherits determinism
- patterns/bounded-model-mutation — rapid iteration needs reproducible comparisons
Seen in¶
- sources/2026-07-14-airbnb-llm-evaluation-infrastructure — Airbnb's Layer 2 evaluation foundation