ByteDance Seed Launches EdgeBench: 12-Hour Long-Horizon Benchmark Shows Frontier AI Models Double Their Learning Speed Every 3 Months

Forum topic · 小凯 · 2026-07-08

Summary

On July 6, ByteDance's Seed team released EdgeBench, a long-horizon AI agent benchmark built from 134 real-world tasks across six domains (science, complex software engineering, knowledge work, algorithm optimization, frontier mathematics, and digital games), with each task supporting 12+ hours of continuous interaction—some exceeding 72 hours. Unlike static benchmarks such as SWE-Bench or HumanEval that measure one-shot correctness, EdgeBench measures how models learn and improve through sustained environmental feedback. Analyzing roughly 38,000 hours of interaction logs and 402 learning curves, the team found that agent environmental learning follows a highly precise log-sigmoid curve (average R² = 0.998), and that frontier models released between September 2025 and May 2026 doubled their environmental learning speed every three months—a new scaling law parallel to pretraining scaling laws. 51 of 134 tasks are open-sourced, with 83 held back to prevent benchmark contamination. The post discusses implications for agent product KPIs shifting from one-shot accuracy to learning slope, plus limitations including small task count and possible domain bias. Links to paper, code, and dataset are included.

On July 6, 2026, ByteDance's Seed team released EdgeBench — a long-horizon benchmark consisting of 134 real-world tasks across six domains, each supporting 12+ hours of continuous work.

Its key difference from traditional benchmarks: instead of measuring "what the model already knows," it measures how a model learns and improves while continuously interacting with an environment.

Scale of the data: 134 real-world tasks, ~38,000 hours of environment interaction logs, 402 learning curves per model, 51 tasks + full framework open-sourced and reproducible.

Core finding: Agent environmental learning follows a highly precise log-sigmoid curve, with average fit R² = 0.998. Frontier models from September 2025 to May 2026 doubled their environmental learning speed every three months.

Links:

Original post: https://seed.bytedance.com/zh/blog/edgebench-%E8%A1%A1%E9%87%8F%E7%9C%9F%E5%AE%9E%E4%B8%96%E7%95%8C%E7%8E%AF%E5%A2%83%E5%AD%A6%E4%B9%A0-%E5%8F%91%E7%8E%B0%E6%96%B0-scaling-law
Project: https://seed.bytedance.com/edgebench
Paper: https://edge-bench.org/paper.pdf
Code: https://github.com/ByteDance-Seed/EdgeBench
Data: https://huggingface.co/datasets/ByteDance-Seed/EdgeBench

What EdgeBench Measures: Running Long Matters More Than Answering Once

Existing benchmarks (SWE-Bench, Terminal-Bench, HumanEval, Aider Polyglot) mostly measure one-shot completion ability and one-shot correctness — judging the model at the moment it's taken out of the box.

EdgeBench asks instead: given enough time, feedback, and room to improve, can a model learn from the environment and keep getting better?

Tasks must therefore satisfy several conditions:

Genuinely sourced from real problems, not synthetic toys
Clear feedback signals: scores, error messages, improvement hints
Long-horizon runnable: they can't finish in seconds

Every task supports at least 12 hours of continuous running, with extended experiments exceeding 72 hours. Human experts average 57.2 hours per task, with a maximum of 320 hours.

Six Domains: "Real" Rather Than Inflated Numbers

1. Science (gravitational wave analysis, materials simulation, computational chemistry, etc.) 2. Complex software engineering (cross-repo bug fixes, dependency upgrades) 3. Knowledge work (report generation, data analysis, process optimization) 4. Algorithm optimization (reverse-engineering algorithms from specific performance bottlenecks, not LeetCode) 5. Frontier mathematics (paper-level conjecture exploration) 6. Digital games (as a controllable proxy testbed)

Key details:

90% of tasks are newly constructed — not migrated from existing datasets. Models can't inflate scores by memorizing problems.
Only 51 of 134 tasks are open-sourced — the other 83 are held back. The benchmark can't be contaminated; it can only be evaluated via API. This is a rigorous practice that aligns with Terminal-Bench and FrontierScience.
Feedback comes from the real environment — not binary right/wrong, but "what specifically went wrong, how badly, and how to fix it." This feedback density makes learning curves meaningful.

Key Numbers: A Log-Sigmoid Curve with R² = 0.998

After running the data, all frontier models' average learning curves converge to a single shape — a log-sigmoid curve — with average R² = 0.998.

The curve means: models don't improve linearly, but follow a slow-fast-plateau pattern — slow at first (understanding the task), accelerating in the middle (mastering feedback patterns), then flattening at the task's ceiling.

The near-perfect fit suggests environmental learning dynamics follow an underlying regularity independent of the specific model — making predictions like "what will this model score in 5 more hours" possible.

Concrete case: In a 12-hour gravitational-wave task run, GPT-5.5 made 247 scored attempts, raising its score from 42.8 to 67.0. This kind of "24-point gain over a long run" is invisible in traditional benchmarks, which only capture the out-of-the-box moment.

The Striking Finding: Learning Speed Doubles Every 3 Months

The Seed team selected 18 tasks with similar initial performance (to control for baseline capability gaps) and measured performance gain within 2 hours as "learning speed" across 6 generations of frontier models released from September 2025 to May 2026.

Conclusion: for the strongest frontier models of their time, learning speed doubles every three months.

In other words, what September 2025's best model learned in 2 hours, March 2026's best model learns in 1. This parallels pretraining scaling laws — but on the "environmental learning" axis.

Extrapolating: if the pace holds, the strongest model in early 2027 would be 4x better at within-2-hour learning than today's — raising the ceiling for embodied AI, long-task agents, and AI Scientist products.

Relationship to Existing Benchmarks: Continuation, Not Replacement

SWE-Bench: one-shot coding correctness, oriented toward single PR fixes
Terminal-Bench: command-line environment operation, oriented toward "shell assistants"
FrontierScience-type benchmarks: depth of scientific reasoning, but still single-submission

EdgeBench measures "can keep improving over a 12-hour run" — which is what "Agent" was supposed to mean, strictly distinguished from "LLM."

Industry implication: after a year of agent companies competing on "+X on SWE-Bench," the target is shifting to "X model's improvement slope within 2 hours on EdgeBench." Next-gen agent KPIs may move from "accuracy" to "improvement slope per unit time."

Limitations and Open Questions

Task count is small. 134 tasks is marginal statistically; widely accepted benchmarks usually need 500+. A second batch may follow.
Possible China-scenario bias. Many tasks may reflect Chinese industry contexts — an advantage for Chinese AI engineers, potentially limiting overseas academic adoption.
Sensitive to model iteration speed. A wave-like release (e.g., GPT-6, Gemini Ultra 3) in late 2026 could break the "doubling every 3 months" curve short-term.
83 withheld tasks cut both ways — they prevent contamination but mean citations require official cooperation channels.

Why This Matters

EdgeBench introduces a new evaluation paradigm: no longer asking "can the model do it," but "can the model learn it."

For AI coding companies: agent product iteration will shift from "SWE-Bench chasing" to improving learning slopes in long-task scenarios.
For embodied AI: the software engineering domain lets robotics teams test whether their code agents can learn over 12-hour runs.
For academia: a log-sigmoid curve with R² = 0.998 suggests environmental learning has its own scaling law — a second curve parallel to pretraining scaling laws, typically a sign of a coming research wave.

The author's prediction: within 6 months, all major AI labs (OpenAI, Anthropic, Google, xAI, Meta) will tune against EdgeBench-style benchmarks — because "learning slope" will become a new product KPI alongside "accuracy."

*Note: This is a translated summary of a Chinese forum post; claims reflect the original author's reporting on the EdgeBench release.*