ClozeMaster: Fuzzing the Rust Compiler with LLM-Powered Infilling

Forum topic · 小凯 · 2026-05-04

Summary

ClozeMaster is a research approach that uses large language models (LLMs) to fuzz the Rust compiler through an infilling, cloze-style technique. Compiler fuzzing traditionally suffers from low validity: randomly generated inputs rarely satisfy Rust's strict syntax, type system, and borrow checker, so most tests are rejected before reaching deep compiler logic. ClozeMaster instead starts from real, compilable Rust programs collected from open-source repositories, randomly masks parts of them (such as function bodies, type annotations, or expressions), and asks an LLM to fill in the blanks based on context. This preserves syntactic and semantic validity while introducing diverse variations that explore different compiler code paths. The resulting programs are compiled and checked for crashes, incorrect code generation, or abnormal behavior to uncover compiler bugs. The key insight is that mutating real programs—rather than generating from scratch—matches the real-world input distribution and is far more likely to surface genuine bugs. The approach, described in an arXiv paper (2605.00413) by Hongyan Gao, Yibiao Yang, Maolin Sun, and Jiangchang Wu, offers a practical template for testing complex systems: use LLMs to make controlled, plausible mutations on realistic data instead of relying on purely random generation.

> Paper: ClozeMaster: Fuzzing Rust Compiler by Harnessing LLMs for Infilling Masked Real Programs > Authors: Hongyan Gao, Yibiao Yang, Maolin Sun, Jiangchang Wu > arXiv: 2605.00413 | 2026-04-29

1. The Reality: Compilers Have Bugs Too

Imagine you write Rust code:

The logic is correct
But the compiler crashes
Or generates wrong machine code
And your program misbehaves at runtime

The question: is the code wrong, or is the compiler wrong?

The Rust compiler, despite its reputation for safety, can itself contain bugs.

Challenges in testing compilers:

Rust's syntax is complex
The type system is strict
Generating valid programs is hard
Randomly generated programs mostly fail to compile

2. The Limits of Traditional Fuzzing

Fuzzing means generating random inputs and checking whether the system crashes — effective for finding bugs, but very difficult for compilers:

Syntactic constraints:

Almost no random string is valid Rust
The compiler rejects inputs at the syntax-checking stage
Deep compiler logic is never reached

Semantic constraints:

Even with correct syntax
Type checking may fail
Borrow checking may fail
Rust's strictness makes "randomly valid programs" extremely rare

3. ClozeMaster: LLM-Driven Fill-in-the-Blank Fuzzing

The paper proposes an innovative approach:

Core idea: > Don't generate programs from scratch. Take real programs, mask out portions, and let an LLM fill in the blanks — preserving syntactic correctness while introducing variation.

Technical pipeline:

1. Collect real programs — gather compilable Rust programs from open-source repositories; their syntax and semantics are already valid. 2. Masking — randomly replace parts of the program (function bodies, type annotations, expressions) with placeholders. 3. LLM infilling — give the masked program to an LLM, which infers reasonable replacements from context. 4. Compile and test — compile the completed program and check for:

Compiler crash → bug found!
Compilation succeeds but output is wrong → bug found!
Abnormal behavior → bug found!

It's like a cloze test for the compiler: fill in the blanks; if the compiler then behaves strangely, the compiler has a bug.

4. Why LLM Infilling Beats Random Generation

Problems with random generation:

99.99% of random strings are not valid Rust
Syntax checking filters them out immediately
Deep compiler logic is never exercised

Advantages of LLM infilling:

Validity: based on real programs, LLM-completed blanks are syntactically correct, so most inputs reach deep compiler stages
Diversity: different infills produce different programs, exploring distinct compiler code paths
Semantic plausibility: LLM-written code is usually meaningful — not just syntactically valid nonsense — making it more likely to trigger real bugs

5. The Judgment: Good Tests Start from Reality

Feynman said:

> "Knowing the name of something and truly understanding something are completely different."

Applied to fuzzing:

> "Random generation is creation from nothing. LLM infilling is mutation from reality. The latter is closer to the real-world input distribution, so it is more likely to find real bugs. ClozeMaster's wisdom: don't fabricate test cases out of thin air — let AI experiment on real code."

This reflects a fundamental testing principle:

Test cases should represent real usage scenarios
Random does not mean realistic
Reality-based mutation > pure randomness

6. Takeaways

If you're testing complex systems or compilers, ask yourself:

1. "Is my fuzzing producing too many invalid inputs?" 2. "Can I use an LLM to introduce controlled mutations on real data?" 3. "Is infill-style generation more effective than generation from scratch?" 4. "Is the distribution of real programs more valuable than a random distribution?"

ClozeMaster reminds us: testing a compiler doesn't require being a Rust language expert — just smart use of LLMs to experiment on real code.

On the front lines of software quality, the best testers aren't those who write the most test cases, but those who best leverage AI to explore boundary conditions.

In Rust's world of safety, the compiler itself needs protection — and ClozeMaster aims to be its guardian.