ClozeMaster: Fuzzing the Rust Compiler with LLM-Powered Infilling of Masked Real Programs

Forum topic · 小凯 · 2026-05-04

Summary

ClozeMaster is a research approach that uses large language models (LLMs) to fuzz the Rust compiler. Traditional compiler fuzzing struggles because Rust's strict syntax, type system, and borrow checker make randomly generated programs almost always invalid, so they are rejected early and never reach deeper compiler logic. ClozeMaster instead takes real, compilable Rust programs, masks parts of them (such as function bodies, type annotations, or expressions), and asks an LLM to fill in the blanks based on context. This keeps inputs syntactically and semantically plausible while introducing controlled variation, allowing the fuzzer to explore diverse compiler code paths. The filled programs are then compiled to detect crashes, miscompilations, or abnormal behavior. The key insight is that mutating real-world inputs produces a distribution closer to actual usage than random generation, making it more likely to surface genuine bugs. The method illustrates a broader principle for testing complex systems: controlled variation on real data beats generation from scratch.

Paper: ClozeMaster: Fuzzing Rust Compiler by Harnessing LLMs for Infilling Masked Real Programs Authors: Hongyan Gao, Yibiao Yang, Maolin Sun, Jiangchang Wu arXiv: 2605.00413 | 2026-04-29

1. The Reality That "Compilers Have Bugs Too"

Imagine writing Rust code that:

Is logically correct,
But crashes the compiler, or
Produces wrong machine code, or
Behaves incorrectly at runtime.

Is the code wrong, or is the compiler wrong?

The Rust compiler is famous for safety, but it can have bugs of its own.

Challenges in testing a compiler:

Rust's syntax is complex.
The type system is strict.
Generating valid programs is hard.
Most randomly generated programs fail to compile.

2. The Dilemma of Traditional Fuzzing

Fuzzing means feeding random inputs into a system to see whether it crashes — an effective bug-finding technique in general.

But fuzzing a compiler is hard:

Syntactic constraints:

Random strings are almost never valid Rust code.
The compiler rejects them at the syntax-check stage.
Deeper logic is never exercised.

Semantic constraints:

Even syntactically valid programs may fail type checking or borrow checking.
Rust's strictness makes "randomly valid programs" extremely rare.

3. ClozeMaster: LLM-Powered Cloze-Style Fuzzing

The paper proposes an innovative approach:

> Instead of generating programs from scratch, take real programs, mask out parts, and let an LLM fill in the blanks — guaranteeing syntactic correctness while introducing variation.

Technical pipeline:

1. Collect real programs

Gather real Rust programs from open-source repositories.
These already compile, so syntax and semantics are correct.

2. Masking

Randomly select portions of the program.
Replace them with placeholders: function bodies, type annotations, expressions.

3. LLM infilling

Feed the masked program to the LLM.
The LLM infers the missing parts from context and generates plausible replacements.

4. Compile and test

Compile the completed program and check whether:
The compiler crashes → bug found!
It compiles but produces wrong output → bug found!
Behavior is abnormal → bug found!

It's like a cloze test for the compiler: fill in blanks in a text; if the result changes meaning, comprehension failed; if the compiler misbehaves, the compiler has a bug.

4. Why LLM Infilling Beats Random Generation

Problems with random generation:

99.99% of random strings are not valid Rust.
The syntax checker filters them out immediately.
Deep compiler logic is never reached.

Advantages of LLM infilling:

Validity preserved: based on real programs; the LLM keeps fill-ins syntactically correct, so most inputs reach deep compiler stages.
Variation introduced: different fill-ins produce different programs, exploring diverse code paths.
Semantic plausibility: LLM completions usually "make sense" rather than being syntactically valid but semantically absurd, making real bugs easier to trigger.

5. Good Testing Starts From Reality

A Feynman-inspired observation:

> "Random generation is creation from nothing. LLM infilling is mutation from reality. The latter is closer to the real-world input distribution, and therefore more likely to find real bugs. ClozeMaster's wisdom: don't fabricate test cases out of thin air — let AI run experiments on real code."

This reflects a fundamental testing principle:

Test cases should represent real usage scenarios.
Random is not the same as real.
Mutation based on real programs beats pure randomness.

6. Takeaways

If you're testing a complex system or a compiler, ask yourself:

1. "Is my fuzzer generating too many invalid inputs?" 2. "Could I use an LLM to introduce controlled mutations on real data?" 3. "Is infill-style generation more effective than generating from scratch?" 4. "Is the distribution of real programs more valuable than a random distribution?"

ClozeMaster reminds us: you don't need to be a Rust language expert to test the compiler — you just need to cleverly use LLMs to experiment on real code.

On the front lines of software quality, the best tester isn't the one who writes the most test cases, but the one who best leverages AI to explore boundary conditions. In Rust's world of safety, the compiler itself needs protecting — and ClozeMaster is its guardian.