Project Zero pits Fuzzilli against BigSleep on V8 bugs

If you are deciding whether to fund an LLM bug-finding agent on top of your fuzzers, Google Project Zero just published the comparison you want. In Fuzzers vs. AI, Samuel Groß measured Fuzzilli, Project Zero's generative JavaScript-engine fuzzer, against BigSleep, an LLM-based bug-finding system, both running in production against new code in V8. The headline: they are complementary, not competing — production overlap was about 25%.

What they measured

Two datasets, both V8-only (the post explicitly leaves JavaScriptCore and other engines to future work):

• Production (April–September 2026): Fuzzilli runs continuously and usually catches a change within hours; BigSleep runs once daily with a 1–2 day delay. Raw reports: Fuzzilli 316, BigSleep 441. After filtering to vulnerability types: Fuzzilli 178, BigSleep 95. After removing duplicates and won't-fix reports, actionable vulnerabilities came to BigSleep 45, Fuzzilli 37. Of 81 actionable reports, 20 overlapped — roughly 25%.
• Experiment: BigSleep was pointed at the commits that likely introduced ~1,150 historic Fuzzilli bugs (reduced to ~600 vulnerabilities, 132 reproducible, 112 after dedup), running 25 attempts per commit with Gemini 3.6 flash. It rediscovered about 62% of the original bugs, found a different bug on ~12% of commits, and found nothing on ~26%. Overlap here was ~60%, rising to ~70% with follow-up runs.

No cost-per-bug and no coverage comparison were attempted — the authors say neither system is calibrated for it. Importantly, they found no statistically significant memorization of pre-2026 bugs.

Where each wins

The split is the useful part:

• The LLM agent leads on bugs needing multi-step logical reasoning, specific "magic" values, new language features, or non-standard harnesses. Groß cites a hole-check-elision variant found by reasoning about labeled breaks, and an integer truncation in exception-handler tables that only triggers on bytecode too large for a fuzzer to stumble onto.
• The fuzzer leads on non-local bugs that need only a few pieces, on bugs hard to spot by reading source (C++ compiler-optimization interactions — e.g. a Maglev bug from two individually correct optimizations), and wherever agent biases (starting point, prompt wording) get in the way. Overlap was notably smaller for JIT-compiler bugs.

Caveats the authors flag

The production overlap is bounded by timing — once a bug is fixed, the other system can no longer rediscover it. The experiment used only bugs that reproduce on a stock x64 build, the BigSleep model was upgraded several times mid-study, and the identified "introducing" commits may be imperfect, so the experiment likely underestimates the agent. The no-memorization finding may not generalize to other models.

Why it matters

This is the first production-scale, apples-to-apples read from a top-tier bug-finding team, and it refuses the hype on both sides: the agent did not replace fuzzing, and fuzzing did not make the agent redundant. Groß's recommendation is blunt — run both — and he expects agents to take a growing share of discoveries as models improve. For defenders triaging a pipeline, the actionable takeaway is to stop treating "AI finds bugs" as a yes/no and start tracking which bug classes your agent adds over your existing fuzzers.