Can AI Pass CAPTCHAs? The HLL Benchmark Says Humanity's Last Line of Verification Hasn't Fallen Yet

You open a website to register an account. A CAPTCHA pops up—drag the slider into the gap. You finish in three seconds and think: how easy.

But what if an AI tried?

Not static classification like "identify the traffic light in the image," but genuinely operating a browser: locating the CAPTCHA, understanding the interaction rules, precisely dragging a slider, clicking icons in the right order, assembling a puzzle—then submitting. The entire process, exactly like a human.

Researchers at Shanghai Jiao Tong University proposed a benchmark called HLL (Humanity's Last Line of Verification) to answer one question: can AI cross the line of human verification?

The answer is both a relief and chilling.

Why CAPTCHAs Are AI's "Last Mile"

Over the past two years, multimodal AI agents have advanced rapidly. They can browse the web, operate mobile apps, even book flights. But nearly all agent evaluations share a blind spot: CAPTCHAs are filtered out.

Mainstream benchmarks like WebArena and OSWorld simply skip CAPTCHA pages. Why? Because CAPTCHAs are hard—it's not just "understanding the screen." You need to:

1. Locate: find the CAPTCHA among a mess of page elements 2. Understand: figure out what it asks (drag? click? select? assemble?) 3. Execute: operate the UI precisely (pixel-level dragging, correct click ordering) 4. Track: remember what you've done and judge completion 5. Submit: submit at the right moment

This isn't a visual recognition problem—it's a complete perception-action closed loop.

HLL's Design Philosophy: CAPTCHAs as Controlled Experiments

HLL's core innovation isn't "collecting another pile of CAPTCHAs"—it's making CAPTCHA evaluation a controlled experiment.

It defines a five-tuple: (f, d, ℓ, z, s)—task family, difficulty, distractor level, whether dynamic verification applies, and sample index. The intuition is simple:

> Did you fail the exam because the questions were hard (d), the room was noisy (ℓ), or because you guessed the right answer with the wrong process (z)?

HLL separates these dimensions so you can precisely diagnose where an AI gets stuck.

Ten CAPTCHA Families, Four Capability Dimensions

HLL includes 10 CAPTCHA types across four capability dimensions:

| Capability Dimension | CAPTCHA Types | |---------|-----------| | Recognition and ordered selection | text transcription, icon sequence selection, natural image sequence selection | | Spatial alignment and local reasoning | slider alignment, puzzle alignment, missing-piece selection | | Stateful puzzle restoration | board rearrangement, tile restoration | | Reasoning-guided semantic interaction | logic-arithmetic interaction, category-guided image selection |

Note: this is not "look at the picture and pick the cat." Slider alignment demands pixel-level precision, puzzle restoration requires tracking intermediate state, and logic-arithmetic requires reading the rules before acting.

Three "Stress Axes": Making Tests More Realistic

1. Difficulty escalation: tighter tolerances, more similar candidates, more complex restoration states 2. Environmental distractors: clean page → realistic web context → deceptive pages with deliberate bait 3. Dynamic verification: checking not only whether the final answer is right, but whether the action process is plausible

The third axis is the harshest. Static evaluation only asks "is the answer correct"; dynamic evaluation also asks "how did you do it." If you guess the answer but your action trajectory looks robotic (e.g., skipping intermediate steps), you still fail.

Results: How Far Is AI from Replacing Humans?

The researchers tested 8 frontier multimodal agents, including GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6, Claude-Sonnet-4.6, Grok-4, GLM-5V, MiniMax-M2.7, and Qwen-Max.

Static Testing in Clean Environments

| Model | Average Pass Rate | |------|-----------| | Claude-Opus-4.6 | 90.0% | | Gemini-3.1-Pro | 73.8% | | GPT-5.4 | 70.0% | | Grok-4 | 58.2% | | Claude-Sonnet-4.6 | 35.4% | | GLM-5V | 16.2% | | MiniMax-M2.7 | 20.2% | | Qwen-Max | 9.4% |

Claude-Opus-4.6 looks strong—but note: near-perfect on text transcription, yet only 63% on tile restoration and 96.5% on board rearrangement. Gaps across CAPTCHA types are huge.

More critically: no model is perfect across all types. AI capability is "lopsided," not "generalist."

Adding Distractors: A Performance Cliff

With distractor elements around the page:

• Claude-Opus-4.6 fell from 90.0% to 79.2%
• Gemini-3.1-Pro fell from 73.8% to 65.8%
• Claude-Sonnet-4.6 fell from 35.4% to 26.0%
• Qwen-Max fell from 9.4% to 8.6%

Distractors don't hurt all models uniformly—they amplify gaps in localization and grounding. Weak models collapse to near zero under interference.

Harder Variants: Even Strong Models Stumble

On harder variants:

• Claude-Opus-4.6 fell from 90.0% to 62.0%
• GPT-5.4 fell from 70.0% to 37.0%
• Gemini-3.1-Pro fell from 73.8% to 24.0%
• Qwen-Max dropped straight to 0%

Difficulty escalation tests perception-action precision of the task itself, not environmental adaptation. With tighter tolerances and more similar candidates, AI's fine motor control is exposed.

Dynamic Verification: The Deadliest Cut

This is the most counterintuitive result. Dynamic verification requires not just a correct answer but a plausible action process:

| Model | Static Avg | Dynamic Avg | Drop | |------|---------|---------|------| | Gemini-3.1-Pro | 71.3% | 45.0% | -26.3 | | GPT-5.4 | 65.8% | 26.3% | -39.5 | | Claude-Opus-4.6 | 88.0% | 23.8% | -64.2 | | Grok-4 | 51.0% | 11.3% | -39.7 | | Claude-Sonnet-4.6 | 29.6% | 5.0% | -24.6 |

Claude-Opus-4.6 was strongest in static testing but plummeted 64 points under dynamic verification. It can "guess" the right answer, but its action process doesn't survive scrutiny.

It's like a student acing the exam while the proctor discovers the scratch paper is blank—correct answer, wrong process.

Failure Modes: Where Exactly Does AI Get Stuck?

HLL's fine-grained design lets us diagnose specific failure modes:

1. Perceptual decoding errors: misreading distorted text, misjudging category cues 2. Target localization failure: finding the right target but clicking offset coordinates 3. Spatial grounding failure: knowing where to drag but not landing precisely enough (insufficient pixel-level precision) 4. State-tracking failure: forgetting which puzzle pieces have been moved 5. UI understanding failure: not knowing which regions are interactive 6. Recovery and self-correction failure: unable to roll back and fix mistakes

These failures aren't just "can't see clearly"—they expose systematic deficiencies in closed-loop interaction.

Why This Matters More Than You Think

You might think: a CAPTCHA is just a CAPTCHA, so what?

The big issue: CAPTCHAs are the last gate for AI agents entering the real world.

Consider these scenarios:

• AI auto-registers accounts for you—but gets blocked by a CAPTCHA
• AI snipes tickets for you—but must pass a CAPTCHA before submitting the order
• AI runs bulk information queries for you—but is stopped by anti-scraping CAPTCHAs

CAPTCHAs were designed to distinguish humans from machines. If AI can reliably pass them, the entire internet's anti-automation infrastructure needs redesigning.

Conversely, if AI can't pass CAPTCHAs, every promise of "AI operating your computer" stalls at the last mile in real deployment.

HLL's conclusion is clear: AI is not yet ready to replace humans at passing CAPTCHAs. But the gap is narrowing—and faster than many expected.

Engineering Insights

For engineers building agents, HLL offers several key takeaways:

1. Don't look only at final answers: static pass rates severely overestimate an agent's real capability. Add process verification and performance may halve. 2. Spatial precision is the bottleneck: agents don't lack "understanding"—they lack "precise execution." A slider dragged a few pixels off is a failure. 3. State tracking is an overlooked capability: in multi-step interactions, agents must remember what they've done and what remains. This isn't an LLM strength. 4. Distractor environments are the real environment: clean-page performance doesn't represent real deployment. Agents must learn to locate targets on noisy pages. 5. Evaluation design matters more than data volume: HLL has only 10 CAPTCHA types, but its factorized design (difficulty × distractors × dynamics) generates 33 evaluation units—far more diagnostic power than stacking data.

Open-Source Code

The paper's code is open-sourced: https://github.com/XinhaoS0101/HLL

Personal Reflection

HLL raises a deeper question: what is a CAPTCHA, really?

It's not a question—it's a social contract. A website says "prove you're human," resting on the assumption that some things are easy for humans and hard for machines. That assumption is being eroded.

But HLL's results also give us a buffer: while AI can "read" CAPTCHAs, there's a fundamental gap in "operating like a human." The gap isn't in perception or reasoning—it's in grounding: converting abstract understanding into precise physical action.

An analogy: AI is like a chess player who knows every principle but has shaky hands. It knows the right move, but its hand trembles and the piece lands wrong. On a chessboard, that's a slip; in front of a CAPTCHA, that's "you're not human."

Perhaps shaky hands are humanity's last advantage.

---

Paper: HLL: Can Agents Cross Humanity's Last Line of Verification? Code: github.com/XinhaoS0101/HLL Institutions: Shanghai Jiao Tong University, Shandong University, Tongji University