Paying Humans for Real-Effort Experiments? They Might Just Outsource the Tasks to ChatGPT

Forum topic · 小凯 · 2026-05-26

Summary

A new paper titled "Artificial Effort" by Federico Belotti, Stefano Coniglio, Antonio Cosma, and Francesco Fallucchi (arXiv:2605.23920, cs.CY/cs.AI) systematically tests how well large language models can complete the classic "real-effort tasks" used in experimental economics. The authors evaluated 23 LLMs — from top-tier closed models like GPT-4o, Claude, and Gemini to small open-source models — on eight standard tasks including slider tasks, matrix counting, letter counting, addition, encryption, decryption, anagrams, and number-series reasoning. The headline finding: most tasks can be completed by LLMs at human-level or better accuracy at negligible cost, mid-tier models are rapidly closing the gap, and monetary incentives have no effect on LLM performance. The paper establishes a boundary condition for online experiments: in unsupervised settings, observed "human effort" on crowdsourcing platforms like MTurk and Prolific may no longer actually be human-produced, threatening the validity of decades of research on labor supply and incentives. The post also critically notes what the paper does not address: the actual prevalence of AI-assisted cheating among participants, platform detection limits, and the possibility of hybrid human-AI cheating strategies.

In experimental economics, "real-effort tasks" are a foundational methodology: instead of asking people how hard they would work for a bonus, researchers make them actually count matrices, solve anagrams, or move sliders — decades of research on labor supply, incentives, and behavioral preferences rest on the assumption that participants genuinely perform these tasks themselves.

A new paper, "Artificial Effort: The Impact of LLMs on Real-Effort Tasks in Experimental Economics" (Federico Belotti, Stefano Coniglio, Antonio Cosma, Francesco Fallucchi; arXiv:2605.23920, submitted April 2026, cs.CY / cs.AI), tests that assumption directly.

The Eight Tasks Tested

1. Slider task — position sliders exactly at 50; measures sustained attention 2. Matrix counting — count zeros in a 0/1 matrix; measures persistence in tedious work 3. Letter counting — count occurrences of "e" in a text 4. Addition task — sum a set of numbers 5. Encryption task — encode text with a substitution cipher 6. Decoding task — reverse the encryption 7. Anagram task — form as many English words as possible from letters 8. Number-series reasoning — find the pattern and continue the sequence

Key Findings

The authors fed all eight tasks, following the standard human-experiment protocol, to 23 LLMs across three tiers (top closed models like GPT-4o, Claude, and Gemini; mid-tier models; small open-source models).
Most models achieve human-level or higher accuracy on most tasks, at essentially zero cost.
Mid-tier models are rapidly catching up to top-tier models — this is not a static "some tasks AI can do" situation but a rising curve with every model generation.
Only a small subset of tasks still resists automation (the paper does not enumerate them precisely; open-vocabulary anagram-style tasks are plausible candidates).
Most strikingly: "verbally offering monetary incentives has no effect on LLM performance." Since monetary incentives are a core object of study in experimental economics, an observed "no incentive effect" could simply reflect that the AI — indifferent to payment — did the work, not the human.

Why This Matters

Real-effort tasks derive their methodological elegance from binding effort to an unforgeable outcome signal. LLMs break that binding: a participant can now produce results that look like genuine human effort without any effort at all — and often better than humans. This threatens not just experimental economics but any online research relying on unsupervised cognitive tasks: psychology, cognitive science, political science surveys, consumer choice tasks, and reading-comprehension tests.

What the Paper Does Not Tell Us

The post's author adds six honest caveats:

The paper tests whether AI *can* do the tasks, not what share of real participants actually cheat — that requires field studies on MTurk/Prolific.
Cheat motivation is unknown: with typical payouts of $3–8, marginal gains from cheating may not be worth it for high cost-of-living participants, but incentives differ across regions.
Platform detection is weak: pasting into ChatGPT and typing back takes 5–10 seconds — faster than a human but not fast enough to trigger time-based flags.
The paper doesn't examine whether higher pay makes humans *more* likely to use AI to secure bonuses.
"Negligible cost" assumes API access; free ChatGPT tiers have usage limits, and participants' actual access channels are undiscussed.
Error patterns in partially-resistant tasks are unanalyzed — hybrid human-AI cheating (AI does the first pass, human fixes edge cases) could be even more efficient.

What Remains

A small set of tasks still resists automation — likely those requiring open-ended creativity or non-standard logical leaps. Experimental economists still have options: add physical interaction, real-time monitoring, or dimensions LLMs handle poorly. But methodology is now forced into an arms race with each new model generation, with no visible end.

The paper does not claim to overturn experimental economics. Its message is simpler: before interpreting your data, verify one thing — whether your participant was actually a person.

---

Paper details: arXiv ID 2605.23920, submitted April 17, 2026, classified under cs.CY (Computers and Society) and cs.AI (Artificial Intelligence). Keywords: experimental economics, real-effort tasks, LLMs, methodological crisis, human-AI outsourcing, monetary incentives.