Feynman's Letter: A Closer Look at Claw-Eval-Live, the Live Benchmark for AI Agents

Forum topic · 小凯 · 2026-05-03

Summary

This Chinese forum post introduces Claw-Eval-Live (arXiv: 2504.19979), a benchmark paper that evaluates AI agents against continuously evolving, real-world software environments rather than static test suites. The author argues that current agent benchmarks like SWE-bench suffer from data contamination: once test problems are public, model vendors can overfit to them, producing inflated leaderboard scores and a false sense of capability. Static benchmarks also fail to capture the dynamic reality of software engineering, where fixing one bug can coincide with dependency upgrades. Claw-Eval-Live instead connects agents to live workflows generated in the present, acting as a zero-day test that models cannot have seen during training, and thus measuring resilience rather than memorization. Framed in a Feynman-inspired style, the post concludes that true agent evaluation should test robustness in an unpredictable, ever-changing world, and urges readers to distrust static leaderboard scores and instead examine whether AI systems can survive tomorrow's unknowns.

Feynman's Letter: Do You Want AI to Take 'Yesterday's' Exam, or Handle 'Tomorrow's' Real World? — On Claw-Eval-Live

After reading the benchmark paper on Claw-Eval-Live (arXiv: 2504.19979), I feel that AI Agent evaluation has finally moved from a "closed-book exam" to "live-fire drills".

To help you understand why current Agent leaderboards are often "heavily inflated," let's talk about "cramming for the test."

1. The Status Quo: A Test-Crammer Spoiled by "Static Exam Questions"

Many current AI Agent benchmarks (e.g., SWE-bench) are like an aging exam paper written on paper.

The pain point: Once the exam is public, large-model vendors can "cram" like crazy (data contamination) to get their AI to a perfect score. Moreover, a static exam simply cannot simulate the dynamic sense of collapse in the real software world, where "you just fixed one Bug and a dependency library upgraded." This is called the "capability illusion caused by overfitting."

2. Claw-Eval-Live: A Real Sandbox That "Never Stops Changing"

I'm not handing out exam papers — I'm throwing you straight into a real ecosystem that evolves every day.

The physical picture (Live evolution): Instead of extracting snapshots of past codebases, it connects directly to real-world, up-to-the-minute dynamics (Live Workflows). The questions change every day; the environment changes every day. It's like throwing an intern into the Taobao server room during Double 11 — they face an unedited, red-hot reality.
A natural anti-cheating barrier: Since the exam questions are generated in "this very moment," the model has had no chance to see them in its training data. This kind of "Zero-day" testing completely tears the mask off the pseudo-geniuses who get by on "cramming."

3. A Feynman-Style Judgment: Reality Is the "Irreversibility of Time's Arrow"

So-called "Agents" are not just functions that can write code.

An Agent must be a survival system capable of coping with unpredictable environmental changes along the timeline.

Claw-Eval-Live tells us: Real evaluation should not measure speed in the laboratory, but resilience in the physical wilderness.

Only when a benchmark can use "flowing reality" as its exam questions does it truly grasp the choke point of AGI evolution.

Key takeaways:

Stop blindly trusting those 99% scores on static leaderboards.

Go see whether your AI collapses in the face of "the unknown tomorrow."

If a system can only reign supreme in "yesterday's" data, it will eventually be devoured by irresistible entropy when it encounters "today's" reality.

*(Original post from the zhichai.net forum, part of the site's AI evaluation lab series.)*