ByteDance Seed Releases EdgeBench: 12-Hour Long-Horizon Benchmark Finds AI Learning Speed Doubles Every Three Months

Forum topic · 小凯 · 2026-07-08

Summary

ByteDance's Seed team has released EdgeBench, a benchmark of 134 real-world tasks across six domains designed to measure how AI agents learn and improve through sustained environmental interaction rather than one-shot problem solving. Each task supports at least 12 hours of continuous work, with some experiments extending past 72 hours; human experts average 57.2 hours per task. The dataset includes roughly 38,000 hours of interaction records and 402 learning curves per model. EdgeBench's central finding: frontier models' environmental learning follows a highly precise log-sigmoid curve with average fit R²=0.998, and between September 2025 and May 2026, the learning speed of the strongest frontier models doubled every three months—a new scaling law running parallel to pre-training scaling laws. Notably, 90% of tasks are newly constructed, and only 51 of 134 tasks are open-sourced to prevent benchmark contamination, with the framework, paper, and a Hugging Face dataset publicly available. The post argues this shifts Agent evaluation from answer accuracy toward improvement rate per unit time.

ByteDance Seed Releases EdgeBench: A 12-Hour Long-Horizon Benchmark for Real-World Agent Learning

> Published 2026-07-06 · ByteDance Seed · AI coding/Agent evaluation > Original post: https://seed.bytedance.com/zh/blog/edgebench-%E8%A1%A1%E9%87%8F%E7%9C%9F%E5%AE%9E%E4%B8%96%E7%95%8C%E7%8E%AF%E5%A2%83%E5%AD%A6%E4%B9%A0-%E5%8F%91%E7%8E%B0%E6%96%B0-scaling-law > Project: https://seed.bytedance.com/edgebench > Paper: https://edge-bench.org/paper.pdf > Code: https://github.com/ByteDance-Seed/EdgeBench > Data: https://huggingface.co/datasets/ByteDance-Seed/EdgeBench

On July 6, ByteDance's Seed team released EdgeBench — a benchmark of 134 real-world tasks across six domains, each supporting more than 12 hours of continuous work.

Its key difference from traditional benchmarks: instead of measuring "what the model already knows," it measures how a model learns and improves through sustained interaction with an environment.

Dataset scale: 134 real-world tasks, ~38,000 hours of environment interaction logs, 402 learning curves per model, 51 tasks plus the full framework open-sourced and reproducible.

Headline finding: Agent environmental learning follows a high-precision log-sigmoid curve, with average fit R²=0.998. From September 2025 to May 2026, frontier models' environmental learning speed doubled every three months.

What EdgeBench Measures: Endurance Beats One-Shot Accuracy

Existing benchmarks (SWE-Bench, Terminal-Bench, HumanEval, Aider Polyglot, etc.) mostly test two things: one-shot task completion ability and the probability of answering correctly on the first try. These matter, but both judge the model in its "out-of-the-box" moment.

EdgeBench asks something different: given enough time, feedback, and room to iterate, can the model "learn" from the environment and keep improving?

For that, tasks must meet several criteria:

Genuinely sourced from real problems, not synthetic toys
Clear feedback signals: scores, error messages, improvement hints
Long-horizon runnability: they can't end in seconds

So every task supports at least 12 hours of continuous operation, with some extended experiments exceeding 72 hours. Human experts average 57.2 hours per task, with a maximum of 320 hours.

Six Domains: Real, Not Just Big Numbers

EdgeBench covers six domains: 1. Science (gravitational-wave analysis, materials simulation, computational chemistry, etc.) 2. Complex software engineering (multi-repo bug fixes, dependency upgrades) 3. White-collar knowledge work (report generation, data analysis, process optimization) 4. Algorithm optimization (not LeetCode-style — deriving algorithms backward from specific performance bottlenecks) 5. Frontier mathematics (paper-level conjecture exploration) 6. Digital games (as a controlled proxy testbed)

Key details:

90% of tasks are newly constructed — not ported from existing datasets. This means tighter quality control and no way for models to score high by "grinding practice sets."
Only 51 of 134 tasks are open-sourced — the other 83 are withheld. The benchmark stays uncontaminated: testing is API-only, and the data can't be downloaded for training. This is a rigorous approach rarely seen in AI evaluation (especially in China's ecosystem) — aligning with the practices of Terminal-Bench and FrontierScience.
Feedback comes from the real environment — not a binary right/wrong, but "exactly what's wrong, how badly, and how to fix it." This feedback density makes learning curves meaningful rather than a dead end where "once correct, nothing changes."

The Key Numbers: A Log-Sigmoid Curve with R²=0.998

After running the data, the team plotted average learning curves for all models. Every frontier model converges to one shape — a log-sigmoid curve — with average R²=0.998 (an absurdly high fit).

The curve means: models don't improve linearly; they follow slow-fast-plateau. Slow at first (figuring out the task), accelerating in the middle (mastering feedback patterns), then flattening (reaching the task's ceiling).

R²=0.998 is meaningful: it implies environmental learning dynamics follow some underlying regularity that is not model-specific — making predictions like "what score will this model reach in 5 more hours" possible.

Concrete example: over a 12-hour run on a gravitational-wave task, GPT-5.5 raised its score from 42.8 to 67.0 across 247 scored attempts. This "24-point gain over a marathon" is invisible on traditional benchmarks — because traditional benchmarks measure only the out-of-the-box moment.

The Sharper Finding: Learning Speed Doubles Every Three Months

The Seed team selected 18 tasks with similar initial performance (to control for baseline capability gaps) and measured six generations of frontier models released from September 2025 to May 2026, using performance gain within 2 hours as the measure of "learning speed."

Conclusion: for the strongest frontier model of each era, this learning speed doubled every three months.

In other words: what the strongest model of September 2025 could learn in 2 hours, the strongest model of March 2026 learns in 1 hour. This parallels pre-training Scaling Laws — but on the "environmental learning" axis.

Pushed further: if the pace holds, the strongest model in early 2027 would have 4× the 2-hour learning capability of today's. For embodied AI, long-task agents, and AI Scientist products — where "time reveals the truth" — the experience ceiling rises with this learning-speed curve.

Relationship to Existing Benchmarks: Continuation, Not Replacement

SWE-Bench: tests one-shot correctness on coding tasks, geared toward one-shot PR fixes
Terminal-Bench: tests command-line environment operation, geared toward "shell assistants"
FrontierScience-style benchmarks: test scientific reasoning depth, but still single-submission

EdgeBench tests "can run for 12 hours and keep improving" — closer to the original meaning of "Agent," strictly distinguished from "LLM."

Industrial implication: for the past year, AI Agent companies have raced to post "+X on SWE-Bench." The endpoint is shifting to "X's improvement rate within 2 hours on EdgeBench" (more stable and trustworthy than one-shot absolute scores). Next-gen Agent KPIs will move from "answer accuracy" to "improvement slope per unit time."

Why a Chinese Team Is Publishing This

For the science, engineering, and white-collar categories among the 134 tasks, Chinese teams have no shortage of real data. The diversity of real tasks is the fundamental reason EdgeBench works — continuing a post-June-26 pattern of Chinese AI teams contributing on evaluation and infrastructure (the RedKnot inference engine, LongCat Owl's all-ASIC training, the ASPIRE robotics framework, Elements Claw's superconductor discovery).

The 51/134 open-source ratio suggests ByteDance kept a "closed zone" — possibly a commercial API evaluation offering later. This resembles Anthropic's approach with Claude safety evaluations: public core data + open evaluation capability + commoditized data-generation pipeline.

Limitations and Open Questions

Task count is still small. 134 tasks is statistically marginal — broad adoption typically requires 500+. A second batch may be coming.
Domain skew toward Chinese scenarios. Many tasks may reflect Chinese industry contexts — an advantage for Chinese AI engineers, but a barrier to overseas academic adoption.
Results are sensitive to model iteration speed. The "double every three months" figure could be briefly shattered if a wave-like release (e.g., GPT-6, Gemini Ultra 3) lands in late 2026.
83 withheld tasks are double-edged — they protect the benchmark but force citation through collaboration channels.

Why This Matters

EdgeBench introduces a new evaluation paradigm: not "can the model do it," but "can the model learn it." A fundamentally different measurement target.

For AI coding companies: Agent iteration over the next two months will shift from "grinding SWE-Bench" to "improving learning slope in long-task settings" — directly tied to long-horizon Agent experience.
For embodied AI: with "complex software engineering" among the six domains, robotics teams can use it to evaluate their robot code agents — not how fast robots run, but whether they "learn something over a 12-hour run."
For academia: a log-sigmoid curve with R²=0.998 implies environmental learning has its own Scaling Law — a second curve parallel to pre-training scaling. A new Scaling Law usually signals a new research wave.

The author's judgment: within six months, every major AI lab (OpenAI, Anthropic, Google, xAI, Meta) will tune against EdgeBench-style benchmarks — not out of patriotism, but because "learning slope" will become a product KPI rivaling "answer accuracy."

*Note: This post was translated/edited from the original Chinese forum post; all figures and claims are as stated by the original author.*