Skip to content

SYSTEM Cited by 1 source

Meta Catching JiTTests

Definition

Catching Just-in-Time Tests (JiTTests) is Meta's production system that uses LLMs to generate executable, diff-specific tests that are designed to fail before a code change lands. A failing test is a concrete demonstration that the diff changed behavior relative to its parent. The system then filters these failures ("weak catches") down to likely real bugs ("strong catches") and contacts the diff author only for high-ranked candidates. It moved from research prototype to part of Meta's infrastructure in about five months. (Source: sources/2026-10-29-meta-catching-bugs-before-they-land-jittestings-human-on-the-loop)

Why it differs from LLM code review

LLM or agentic code review (for example systems/cloudflare-ai-code-review) reads a diff and predicts faults, like a faster human reviewer. JiTTests instead executes generated tests and reports an observed behavioral difference. It needs more compute steps but catches bug classes review-style tools can miss, and it turns the human question from "is this comment right?" into "was this behavior change intended?".

Pipeline

  1. Targeting and triggering. Runs overnight on spare capacity (concepts/opportunistic-capacity-harvesting). Selects the previous day's diffs that a risk model — similar to Meta's Diff Risk Score and trained on past changes — marks as risky.
  2. Two test-generation workflows.
  3. Dodgy diff: treats the diff itself as a mutant/bug and uses mutation-guided LLM test generation to produce tests that distinguish diff from parent, without reasoning about intent. Goal: maximize the weak-catch pool.
  4. Intent-aware: an LLM infers the diff's intent from code, title, and summary; describes "risks" (ways the implementation could go wrong); builds a mutant per risk; generates tests that kill each mutant; runs those tests on the actual diff. Failures are harvested as weak catches.
  5. LLM-as-Judge ensemble. Several SOTA LLMs classify each weak catch using the test code, the execution outcome (failure code, message, trace), the diff, and the inferred intent.
  6. RubFake (see below) adds deterministic judgments.
  7. Human-on-the-loop outreach. The combined ranking decides whether to contact the engineer. Engineers set criteria and review outcomes in aggregate (concepts/human-in-the-loop).

RubFake (Rule-Based False-Positive Killing Environment)

RubFake is the rule-based assessor that sits next to the LLM judge ensemble. Meta's rationale: the LLM judges have rich semantics but are non-deterministic and hard to verify; RubFake gives verified, deterministic judgments. The two together produce one ranking of weak-catch candidates. This is the patterns/deterministic-tool-vs-llm-judgment split applied to false-positive filtering. The post does not list RubFake's rules.

Production impact (as of 2026-10)

  • 589 engineers engaged — 64% of outreach.
  • 60% true positives reported by engaged engineers.
  • 74 SEV-level privacy bugs caught at diff stage in H1 2026; privacy SEV occurrence down 20% in that period.

Design lessons

  • High-recall generation (two workflows) plus a precision filter (ensemble + rules) is how the system keeps a usable true-positive rate.
  • Human attention is the scarce resource; the outreach threshold exists to avoid pulling engineers in for weak catches (concepts/alert-fatigue).
  • Keep it off the developer latency path: batch, risk-targeted, spare capacity.

Seen in

Last updated · 854 distilled / 2,448 read