Summary
A new paper (arXiv 2610.31636) by Marek Šuppa, Ivan Vykopal, and Jaroslav Kopčan proposes using LLM agents as corpus debuggers rather than classifiers to audit multilingual web corpora. A tool-using agent explores the Slovak portion of FineWeb2 (26.5 million documents) through a read-only, budgeted API, compiles findings into executable tests, and has native speakers validate them. Three runs, each costing only $2.21-$2.84, surfaced seven findings across four phenomena. The largest: a machine-generated content farm of 1,408,437 documents (5.32% of the corpus) that passed the pipeline's filters, with 86.6% also surviving stricter thresholds. Blind annotation confirmed six of seven findings (defect rates .53-1.00 vs. .08 for random controls). The compiled test suite scans the full corpus in 0.7 seconds without a GPU, versus $12,000-$90,000 per LLM classification pass. The framework, Slovak test suite, validation data, and agent trajectories are released publicly.
Paper Overview
- Field: NLP
- Authors: Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan
- Published: 2026-10-22
- arXiv: 2610.31636
Abstract
Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. This work uses an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API, compiles each finding into an executable test, and a native speaker then validates the results.
Key Findings
- On the 26.5M-document Slovak portion of FineWeb2, three runs costing $2.21-$2.84 each found seven findings across four phenomena.
- The largest finding is a machine-generated content farm of 1,408,437 documents (5.32% of the corpus) that passed the pipeline's filters; 86.6% of these documents also passed stricter threshold settings.
- Blind annotation supported six of the seven findings, with defect rates of .53-1.00 compared to .08 for random control samples.
- The compiled test suite scans the full corpus in 0.7 seconds without a GPU, versus $12k-$90k per LLM classification pass.
Release
The authors release the auditing framework, the Slovak test suite, validation data, and agent trajectories.
Source
- arXiv: 2610.31636
*Collected automatically on 2026-10-24.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636418