Skip to content

META 2026-10-29

Read original ↗

Meta — Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale

Summary

A 2026-10-29 Meta Engineering post that reports the production deployment of Catching Just-in-Time Tests (JiTTests). JiTTests is an LLM-based system that generates executable tests for each code change (diff) before the diff lands. The tests are designed to fail: a failing test shows a concrete behavioral difference that the diff introduced. Meta frames this as fundamentally different from LLM code review, which reads the diff and guesses at faults like a faster human reviewer. JiTTests costs more compute but catches bug classes that review-style tools can miss. The pipeline has five stages: (1) an overnight batch on spare capacity targets the previous day's "risky" diffs with a Diff-Risk-Score-like model; (2) two test-generation workflows run in parallel — a "dodgy diff" workflow that treats the diff itself as a mutant, and an intent-aware workflow that infers the diff's intent, lists risks, builds mutants from those risks, and generates tests to kill them; failing tests become weak catches; (3) an LLM-as-Judge ensemble of several SOTA LLMs classifies weak catches; (4) a deterministic rule-based assessor, RubFake, adds verified judgments; (5) the combined ranking decides whether to contact the diff author — a human-on-the-loop model where engineers set criteria and review outcomes in aggregate, not each output. Results: 589 engineers engaged (64% of outreach), a 60% true-positive rate, and in H1 2026 74 serious (SEV) privacy bugs caught at diff time, which lowered the occurrence of privacy SEVs by 20%.

Key takeaways

  1. Execute, don't just read. The core design choice is to produce an executable test that shows a behavioral difference between the diff and its parent, not a natural-language review comment. A failing test is evidence; a review comment is an opinion. Meta accepts more compute steps in exchange for catching bugs that LLM or agentic code review can miss. (Source: this article; contrast with systems/cloudflare-ai-code-review)
  2. Two complementary generators maximize recall, then a filter fixes precision. The dodgy diff workflow assumes every diff is buggy and uses mutation-guided LLM test generation to distinguish diff behavior from parent behavior — its goal is a large pool of weak catches. The intent-aware workflow reasons about what the diff tries to do, writes "risks", turns each risk into a mutant, and generates tests that kill the mutant; those tests then run on the real diff, and failures are harvested. High-recall generation plus a strict downstream filter is the shape. (Source: this article)
  3. Weak catch vs strong catch is the central problem. At Meta's scale (hundreds of millions of lines of code) the hard part is not generating failing tests — it is telling true-positive failures (real bugs) from false positives without slowing engineers down. (Source: this article)
  4. Ensemble judge + deterministic rules. The LLM-as-Judge ensemble reads the test code, the execution outcome (failure code, message, trace), the diff, and the inferred intent. Meta calls its judgments semantically rich but non-deterministic and hard to verify, so it pairs the ensemble with RubFake, a rule-based assessor that gives verified, deterministic judgments. Both feed one ranking. This is a clean instance of patterns/deterministic-tool-vs-llm-judgment. (Source: this article)
  5. Human-on-the-loop, not human-in-the-loop. A human does not review every output. Engineers set criteria and review outcomes at an aggregate level; the system contacts a diff author only when the ranking is high enough. The author then only confirms whether an observed behavioral change was intended — they "rarely need to read test code." The explicit goal is to keep human labeling consistent while reducing the chance that an engineer is pulled in for a weak catch (see concepts/human-in-the-loop, concepts/alert-fatigue). (Source: this article)
  6. Risk-targeted, off-peak batch execution. The system does not run on every diff in the critical path. It runs overnight on spare capacity against the previous day's diffs that a risk model (similar to Meta's Diff Risk Score, trained on past changes) marks as risky. This bounds compute cost and keeps it off the developer's latency path — a form of concepts/opportunistic-capacity-harvesting. (Source: this article)
  7. Measured production impact. 589 engineers (64% of those contacted) engaged; engineers report 60% true positives; 74 SEV-level privacy bugs were caught at the diff stage in H1 2026, a 20% drop in privacy SEV occurrence over that period. (Source: this article)
  8. Generalizable operating model. Meta's stated lesson for AI-executed work: define success criteria early, let AI execute, monitor outcomes in aggregate, and intervene at the policy level rather than per item. (Source: this article)

Architecture (five stages)

Stage What it does Determinism
1. Targeting & triggering Overnight on spare capacity; select previous-day diffs ranked risky by a DRS-like model trained on past changes model-scored
2a. Dodgy-diff workflow Treat the diff as a mutant; mutation-guided LLM test generation to distinguish diff from parent; maximize weak-catch pool LLM
2b. Intent-aware workflow Infer intent from code + title + summary → describe risks → build mutants per risk → generate tests that kill each mutant → run on actual diff → harvest failures LLM
3. LLM-as-Judge ensemble Several SOTA LLMs classify each weak catch using test code, execution outcome, diff, inferred intent non-deterministic
4. RubFake Rule-Based False-Positive Killing Environment; verified rule judgments deterministic
5. Human-on-the-loop Combined ranking decides whether to contact the engineer; humans set criteria and review in aggregate policy

Operational numbers

  • 589 engineers engaged — 64% of outreach.
  • 60% true-positive rate reported by engaged engineers.
  • 74 SEV-level privacy bugs caught at diff stage, H1 2026.
  • 20% reduction in privacy SEV occurrence over the same period.
  • "Five months" from first public unveiling to full deployment.

Systems / concepts extracted

Caveats

  • First-party results post; the methodology lives in the FSE 2026 paper ("Just-in-Time Catching Test Generation at Meta", arXiv 2601.22832), which is not ingested here.
  • Table 1 (LLM review vs JiTTests), Table 2 (human-in vs human-on the loop), and the architecture/impact figures are images and are not transcribed.
  • The 60% true-positive rate is engineer-reported among engaged engineers, not a measured precision over all outreach.
  • No detail on RubFake's rules, the judge ensemble's models or voting rule, the ranking threshold, compute cost, or the risk model's features.
  • The 20% privacy-SEV reduction is attributed with "played a key role" wording; the causal share is not quantified.
  • Earlier JiTTests / mutation-testing posts (2025-02 ACH, 2025-09) are linked but not ingested in this wiki.

Source

Last updated · 854 distilled / 2,448 read