Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors

Field: NLP Authors: Madhav S. Baidya, S. S. Baidya, Chirag Chawla Published: 2025-03-18 arXiv: 2503.13843

Summary

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness.

This paper proposes a comprehensive benchmark that evaluates multiple detection methods on two corpora:

• HC3: 23,363 human-ChatGPT pairs
• ELI5: 15,000 human-Mistral-7B pairs

Methods evaluated include:

• Classical classifiers
• Fine-tuned Transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3)
• CNNs
• XGBoost-style stylometric models
• Perplexity-based detectors
• LLM-as-detector prompting

Key Findings

• Transformer models achieve near-perfect in-distribution performance but degrade under domain shift.
• XGBoost-style stylometric models match Transformer performance while retaining interpretability.
• LLM-based detectors perform poorly, affected by generator-detector identity bias.
• Perplexity-based methods exhibit polarity inversion: modern LLM outputs show lower perplexity than human text, but the approach remains effective after correction.
• No single method generalizes robustly across both domains and LLM sources.

The benchmark highlights that current AI-generated text detection remains an open problem, particularly for out-of-distribution settings and novel generator models.

--- *Originally posted on zhichai.net. Paper auto-collected 2026-03-19.*