The problem with tests as they stand: a behaviour bug is found either by a reviewer reading a diff, or by a red suite days after the change landed. Both arrive late. JiTTesting moves the red earlier by writing a throwaway test for the diff while it is still a merge candidate, running it against the parent commit and against the changed code, and alerting the author only when those two results disagree.

The catch is that a disagreement proves almost nothing on its own. A probe that passes on the parent and fails on the diff has shown that the code changed behaviour, and most of the time changing behaviour is the entire point of the diff. A renamed field, a new default, a tightened validation rule, a corrected error message: each one goes red for the same reason a bug does. The probe was also written by a model that read the same diff, so it can encode a wrong expectation of the new behaviour, pass on the parent, and look identical to a real catch. The remaining two outcomes carry even less: a probe that fails on both trees is testing the environment, and one that passes on both noticed nothing. Most generated probes land in one of those three cells, so the raw catch count says very little about how many bugs are in the diff.

That is why the design is a funnel rather than a detector. Meta runs two workflows, dodgy diff (aim at every change) and intent-aware (infer the goal, then aim at the ways it could fail), pools their catches, and drops nearly everything through two filters before an engineer sees one. The filters are what make the numbers work: 60% of the catches that survive to an engineer were confirmed true positives in H1 2026, and the human is asked one question, was this behaviour change intended. The funnel has its own failure mode, though. Intent-aware can infer the goal wrongly, build mutants the diff already passes, kill every probe it generated, and report a clean run with no signal that anything went wrong. The cards below carry that picture end to end: the two workflows, the comparison, the filter funnel and the operating rules. Card E is the one to read first if you only read one, because it shows what happens to a generated test.

A

Two workflows, one kind of test

The LLM writes the same artifact in both cases: a test that passes on the parent commit and fails on the changed code.

What changes is the target. The dodgy diff already has a bug to aim at, because it treats the diff as the bug. Intent-aware has to invent the bug from an inferred goal.

So the answer to "two types of tests" is no. One test type, two ways to aim it.

B

Dodgy diff, anomaly-driven

  • No intent pass and no risk model, so it is the cheaper workflow.
  • It aims at every behaviour change the diff makes, intended or not.
  • It cannot tell a deliberate behaviour change from a defect, so it produces many catches that a human would reject.
  • Its job is recall. It fills the pool of weak catches, on the bet that a real bug sits in there.
C

Intent-aware, hypothesis-driven

  • The LLM reads the diff, its title and its summary, then states what the change was trying to do.
  • It lists the ways that goal could be implemented wrongly. Each risk becomes one synthetic mutant.
  • Tests must kill those mutants, which means pass on the parent and fail on the mutant.
  • Behaviour that contradicts the author's own stated goal is strong evidence of a bug. That is the point of the extra pass.
D

Mental model and where they differ

  • Dodgy diff is anomaly-driven testing: differential, mechanical, high recall, no understanding. The closest classical relative is a regression or differential test generator, or a fuzzer aimed at the diff.
  • Intent-aware is hypothesis-driven testing: infer the goal, enumerate the ways it fails, assert against those.
DimensionDodgy diffIntent-aware
AssumptionThe diff is faultyThe diff has a goal, met badly
Target bugThe diff itselfOne mutant per inferred risk
Extra inputNoneTitle, summary, inferred intent
RecallHigher, it sees every changeLower, it sees only inferred risks
Precision before filteringLower, most behaviour changes are intendedHigher, each catch maps to a named risk
CostLowerOne extra LLM pass
E

What happens to the tests

"Diff" means the changed code, the version that would merge. "Parent" is the code before the change. A useful probe passes on the parent and fails on the diff.

Two aims, one funnel Dodgy diff and intent-aware probes converge into one useful probe, which passes on the parent and fails on the diff. Every other outcome is dropped. Two filters then rank the survivors, and the engineer confirms whether the behaviour change was intended. FIGURE 1 TWO AIMS, ONE FUNNEL Dodgy diff aims at the whole diff Intent-aware aims at inferred risks Useful probe parent pass, diff fail Two filters Judge, then RubFake Engineer confirms intent Not a catch no diff, flaky, new behaviour

Only the top branch earns an alert. The funnel, not the generator, is what turns a flood of behaviour differences into a 60% true-positive rate at the engineer's desk.

LLM-as-JudgeAn ensemble of several strong LLMs, each classifying a catch as a likely true or false positive. Reads meaning, repeats itself poorly.
RubFakeRule-Based False-Positive Killing Environment: fixed rules with verifiable output. Narrow, but deterministic.
ParentDiffWhat it meansKept
passfailThe change broke behaviour the parent hadweak catch
passpassNo behavioural difference founddropped
failfailThe test is flaky or environment-dependentdropped
failpassThe change adds behaviour the parent lackednot bug evidence alone
  • The red result is the evidence. It arrives on the merge candidate, before the change lands, so no engineer ever sees a red suite.
  • These probes never join the repository suite. Their job ends when the engineer confirms the bug or declares the change intended.
  • Of the catches that reach an engineer, 60% are confirmed true positives. Meta gives one figure for both workflows.
F

How to run it

  1. Run at night on spare capacity, over the previous day's diffs.
  2. Rank the diffs first with a risk score. Only risky diffs get tests.
  3. Run both workflows on the same diff. Pool the weak catches.
  4. Filter with LLM-as-Judge and RubFake. Reach out only above a threshold.
  5. Watch the true-positive rate in aggregate. Tune thresholds, not single catches.

This is human-on-the-loop. The engineer usually confirms one thing: was this behaviour change intended?

G

What it depends on

The points below are my reading, not claims from Meta.

  • Weak intent inference gives mutants that the diff already passes. The run ends with no catch and no signal, and nobody knows the intent was wrong.
  • RubFake is deterministic and cheap to trust. The LLM judges drift, so their thresholds need watching.
  • Two workflows cost double the compute. Risk-based targeting, and spare overnight capacity, are what keep it affordable.

Catching Bugs Before They Land (Meta Engineering) · Engineering at Meta