CONCEPT Cited by 1 source

Per-sample evaluation cache

Definition

A caching strategy for LLM evaluation that keys cached results at the individual sample level on two axes:

1. Reference cache — keyed by (sample_identifier, reference_generation_config). Prevents reference regeneration noise.
2. Judge score cache — keyed by (sample, model_output, judge_config, metric). Prevents judge drift noise.

Identical inputs return cached results, making evaluation deterministic, efficient, and comparable across runs. This is simpler and more effective than probabilistic approaches (majority voting, Bayesian noise modeling) because it eliminates noise at the source rather than modeling around it.

Key properties

• Reproducibility: same inputs → same scores, across runs and across time
• Resumability: partial progress is durable — a job failing at example 8,000 resumes from cache
• Incrementality: new candidates/metrics evaluate against existing cached outputs for free
• Efficiency: over half of model outputs across candidates are identical strings — caching eliminates redundant computation

Why not majority voting or Bayesian approaches?

• Majority voting converges toward the judge's central tendency, not toward accuracy
• Bayesian methods need a centralized store for priors/posteriors — effectively the same infra as a cache but without the reproducibility benefit
• Stabilizing inputs is simpler than modeling output instability

Seen in

• sources/2026-07-14-airbnb-llm-evaluation-infrastructure — Airbnb's Layer 2 evaluation foundation; caching on both reference and judge axes