CEO-Bench: Princeton's 500-Day Startup Simulation Finds Only 3 of 14 AI Agents Turn a Profit

Forum topic · 小凯 · 2026-06-29

Summary

Princeton researchers introduced CEO-Bench, a benchmark where AI agents run a simulated subscription software company, NovaMind, for 500 in-simulation days starting with $1 million in cash. Across 14 tested models, only three exceeded their starting capital in their best runs: Claude Fable 5 ($47.15 million), Claude Opus 4.8 ($27.8 million), and GPT-5.5 ($21.3 million). Strikingly, a simple rule-based heuristic that never calls a language model earned $15.76 million, beating every model except the top three. The paper (arXiv 2606.18543) attributes most failures not to weak coding ability—most models wrote valid commands and SQL—but to inability to sustain coherent strategy over long horizons. Successful agents showed proactive exploration, hidden-information mining, cash-flow forecasting, and rapid adaptation. The benchmark marks a shift from single-task evaluation toward long-horizon strategic decision-making, though the authors note limitations: NovaMind is a simplified SaaS environment, the theoretical ceiling is roughly $2.2 billion, and results for the unreleased Claude Fable 5 require independent replication.

Princeton researchers have released CEO-Bench, a new benchmark (arXiv 2606.18543) that tasks AI agents with running a fictional subscription software company, NovaMind, for 500 simulated days starting with $1 million in cash. Of the 14 models tested, only three exceeded their starting capital in their best runs.

Key facts

Researchers: Princeton (Chen, Narasimhan, Liu); reported by The Decoder on June 28, 2026
Environment: NovaMind simulated SaaS company — 500 simulated days, 34 Python API tools, 19 database tables, 26 customer segments, a simulated social network, and hidden variables (satisfaction, willingness to pay, minimum quality expectations)
Failure condition: cash below zero means bankruptcy
Results:
Claude Fable 5: $47.15M (only model to exceed starting capital in multiple runs)
Claude Opus 4.8: $27.8M
GPT-5.5: $21.3M
Rule-based heuristic (no LLM): $15.76M — beat all models except the top three
Theoretical ceiling: roughly $2.2 billion

Why most models fail

The paper's key finding: most models generate valid commands and SQL queries but cannot sustain a coherent strategy. Single-step execution capability is 80%+; long-horizon strategic consistency is 30–40%. This echoes recent industry observations that agents can "do the task" but struggle to "keep doing it right over time."

The heuristic problem

The most striking result is that a simple rule-based heuristic — fixed pricing, targeted ads and development toward key segments, capacity adjusted to recent usage — reached $15.76M. It wins through stability, interpretability, and zero randomness, while LLMs win through exploration and adaptation. In environments without continuous anomalous signals, the value of stability is underestimated.

Behavioral findings

Opus 4.8 and GPT-5.5 proactively explored, testing new strategies for acquisition, tiers, and support budgets
Opus 4.7 was passively conservative — cut costs when hurt, survived but did not grow
Surprisingly strong coding: Opus 4.8 wrote internal simulations to forecast cash flow; GPT-5.5 mined negotiation-history databases for hidden customer preferences

Four measurable success factors: hidden-information mining, forecasting (4-week cash prediction error), rapid adaptation, and proactive planning. The top two models scored above average on all four.

Tooling side effects

Counterintuitively, Claude Opus 4.7 paired with Claude Code and GPT-5.5 paired with Codex performed worse — the authors suspect software-development-tuned system prompts made agents act less frequently and more conservatively. Tool choice shapes agent behavior in ways vendors may not realize.

Why it matters

A post-SWE-bench agent benchmark is forming: CEO-Bench separates agents that "can code" from agents that "can run a company"
Enterprise agent frameworks face a hard question: many integrations may still lose to a stable heuristic
For founders and investors, "sustained correctness" appears more valuable than short-term brilliance — most agents cannot manage even 500 days of robust decisions

Caveats

1. NovaMind is a simplified SaaS; compliance, security, fundraising, and board dynamics are not modeled 2. The huge gap between the $2.2B ceiling and best result ($47.15M) means optimal strategies remain undiscovered 3. Claude Fable 5 is an unreleased Anthropic model; one run aborted and some requests fell back to Opus 4.8, so independent replication is needed 4. No Chinese models (Qwen, DeepSeek, GLM, Kimi, etc.) were tested, and no human-agent collaboration setting was evaluated

CEO-Bench pushes agent evaluation from "solving problems" to "running a company." The 14 models, 500 days, 3 profitable runs ratio says more than any single score.