$2.84 to Audit 26.5 Million Documents: Using an LLM Agent as a Corpus Debugger

A $2.21 Discovery

FineWeb2 is one of the largest open-source pre-training corpora available today, with web text cleaned from Common Crawl across more than 1,000 languages. Its Slovak portion contains 26.5 million documents.

The corpus's cleaning pipeline follows a "one recipe for a thousand languages" approach — it works well for English, but what about Slovak? Nobody had specifically checked.

Marek Šuppa and his team did something clever: they used an LLM as a one-shot corpus debugger. Not classifying documents with an LLM one by one (far too expensive), but letting a tool-using agent explore the corpus through a read-only API, and after discovering systematic defects, compiling each finding into an executable test script.

Three runs, each costing $2.21 to $2.84.

What did they find? 1.4 million machine-generated spam documents — 5.32% of the entire Slovak corpus. These content-farm documents passed all of FineWeb2's cleaning filters, and 86.6% even passed stricter thresholds.

Method: The Agent Is a Debugger, Not a Classifier

This is the paper's core differentiator:

Traditional approach: have an LLM label each document as "machine-generated or not." For 26.5 million documents, at ~$0.001 per document, that's $26,000 — $12k to $90k per pass.

This paper's approach: use an LLM once, as a debugger. The agent explores the corpus via a read-only API (sampling, searching, computing statistics), and once it finds a pattern, compiles the finding into an executable test script. That script scans all 26.5 million documents in 0.7 seconds, with no GPU needed.

Analogy: the traditional approach is a quality inspector checking every part by hand; this paper has the inspector write a detection rule, then lets a machine run it.

Key design choices:

• Read-only, budget-capped API: the agent can only look, not modify; API calls are budget-limited
• Compilation into tests: every finding becomes a reproducible check script
• Native-speaker validation: patterns found by the agent are confirmed by blind review from Slovak native speakers
• No taxonomy leakage: the agent isn't told "what to look for" — it explores on its own

Results: 7 Findings, 6 Confirmed

Across three runs, the agent found 7 systematic defects spanning 4 categories of phenomena. The largest was the 5.32% machine-generated content farm.

Blind review results: 6 of 7 findings confirmed by native speakers (defect rates 0.53–1.00; a random control group scored only 0.08). One finding was not supported.

The compiled test suite scans the full corpus in just 0.7 seconds — versus $12k–$90k per LLM-classification pass.

But the paper honestly reports a counterintuitive result:

> The best static-sampling baseline validated more selectors than any single agent run, at less than half the cost.

In other words, if you only want "find as many defect types as possible," static sampling beats the agent. So where's the agent's value?

The Agent's Unique Contributions

The paper identifies four contributions the agent makes that static sampling cannot:

1. Precise prevalence: not just "this problem exists," but "1.4 million documents, 5.32% of the corpus" 2. Precision estimation: the agent samples its own findings and reports a precision range 3. Root-cause analysis: the agent explains "why the pipeline missed this class of problems" 4. Reusable tests: the compiled scripts work permanently, with no LLM dependency

Static sampling can "discover" problems, but it won't tell you how big they are, why they exist, or how to detect them forever.

A Deeper Insight

The paper's most thought-provoking design choice: the agent runs only once.

Not "an agent continuously monitoring the corpus," but "the agent runs once, produces a test suite, and the test suite takes over." This is fundamentally different from an "LLM classifier":

• LLM classifier: costs money on every pass — $12k–$90k for 26.5 million documents each time
• Agent debugger: $2.84 once, produces a 0.7-second test script, free to run forever after

This points to a more general pattern: an agent's best use may not be continuous operation, but one-shot production of reusable tools. The agent is a tool-maker, not the tool itself.

Honest Assessment

Limitations of the paper:

• Full experiments were run only on Slovak; applicability to other languages is unknown
• 7 findings seems few (though the largest affects 5.32% of the corpus)
• Static sampling actually beat the agent on "number of defect types found" — the agent's advantage is depth, not breadth
• Agent output quality varies run to run; the three runs differed

But as a proof of concept for "agents in data quality auditing," the paper's design is elegant. It clearly delineates what agents are good at and what they aren't — rather than blindly claiming "agents can do anything."

---

Paper: arxiv.org/abs/2610.31636 Code: github.com/NaiveNeuron/agentic-corpus-debugger