Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors

Forum topic · 小凯 · 2026-03-19

Summary

A 2025 arXiv paper (2503.13843) by Madhav S. Baidya, S. S. Baidya, and Chirag Chawla presents a comprehensive benchmark of machine-generated text detection methods. The study evaluates classical classifiers, fine-tuned Transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), CNNs, XGBoost-style stylometric models, perplexity-based detectors, and LLM-as-detector prompting on two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Key findings: Transformer models achieve near-perfect in-distribution performance but degrade under domain shift; XGBoost-style stylometric models match performance while remaining interpretable; LLM-based detectors underperform due to generator-detector identity bias; perplexity-based methods exhibit polarity reversal, as modern LLM outputs show lower perplexity than human text, though they remain effective after correction. No single method generalizes robustly across domains and LLM sources, highlighting open challenges in AI text detection.

Paper Overview

Research Area: NLP Authors: Madhav S. Baidya, S. S. Baidya, Chirag Chawla Published: 2025-03-18 arXiv: 2503.13843

Abstract

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating a diverse set of detection approaches on two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned Transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), CNNs, XGBoost-style stylometric models, perplexity-based detectors, and LLM-as-detector prompting.

Key Findings

Transformer models achieve near-perfect in-distribution performance but degrade under domain shift.
XGBoost-style stylometric models match Transformer performance while remaining interpretable.
LLM-as-detector approaches underperform, hampered by generator-detector identity bias.
Perplexity-based methods exhibit polarity reversal: modern LLM outputs show lower perplexity than human text, though they remain effective after correction.
No single method generalizes robustly across domains and LLM sources.

Significance

This benchmark reveals fundamental limitations of current AI-generated text detection approaches and underscores the need for detectors that generalize across domains, LLM generators, and adversarial conditions.