CEO-Bench: 500 Days Simulating a Startup — Only 3 of 14 AI Agents Beat Their Starting Capital

TL;DR: Princeton researchers built CEO-Bench, where AI agents run a fictional subscription software company, NovaMind, for 500 simulated days (starting capital: $1 million). Of 14 tested models, only Claude Fable 5 (best run: $47.15M), Claude Opus 4.8 ($27.8M), and GPT-5.5 ($21.3M) turned a profit above starting capital in their best rounds. A simple rule-based heuristic that never calls an LLM reached $15.76M — beating all but the top 3 models. Paper: arXiv 2606.18543, reported by The Decoder on June 28, 2026.

The Benchmark

• Researchers: Princeton (Chen, Narasimhan, Liu)
• Environment: NovaMind, a simulated subscription SaaS company; 500 simulated days; 34 Python API tools; 19 database tables
• Initial conditions: 0 customers, $1M cash; cash below $0 = bankruptcy
• Models tested: 14
• Results:
• Only 3 models exceeded starting capital in their best runs
• Claude Fable 5: $47.15M (the only model to exceed it multiple times)
• Claude Opus 4.8: $27.8M
• GPT-5.5: $21.3M
• Rule-based heuristic (no LLM): $15.76M
• Theoretical ceiling: ~$2.2B

Layer 1: A Shift in Evaluation Paradigm

CEO-Bench moves evaluation from single-task success rates to long-chain strategic decision-making:

• Delayed feedback: Revenue arrives only on billing days; R&D projects take days to weeks; mistakes surface as customer churn
• Hidden variables: Customer satisfaction, willingness to pay, minimum quality expectations — agents must infer these from cancellations, support tickets, and social network reactions
• Continuous change: Competitors raise expectations, preferences drift, and simulated business cycles affect demand
• 26 customer segments + simulated social network: Decisions under multi-agent, noisy signals

Unlike SWE-bench, AIME, or HumanEval — which test *whether a model can do something* — CEO-Bench tests *whether it can keep doing the right thing for 500 days*.

Layer 2: Why Most Models Went Bankrupt

The authors highlight a key phenomenon: most models generate valid commands and SQL queries but cannot sustain a coherent strategy. Roughly:

> Single-step execution capability: 80%+; long-chain strategic consistency: 30–40%

This echoes findings shared by ByteDance's Hong Dingkun at Volcano Engine Force (June 23): single-run correctness above 80%, but deliverability only 40–60. Cursor's SWE-bench Pro audit (June 26) pointed the same direction — 63% of "successful fixes" came from retrieval gaming. The core bottleneck of agent engineering is shifting from "can it do it" to "can it keep doing it right."

Layer 3: Why a Rule Heuristic Beat Most LLMs

The most striking result: a simple heuristic with no language model calls reached $15.76M. Its strategy is plain:

• Fixed pricing, quotas, and subscription tiers
• Targeted ads and development for key customer segments
• Capacity adjustments based on recent usage

It wins on stability, interpretability, and zero randomness. LLMs win on exploration, adaptation, and learning from anomalous signals. CEO-Bench's finding: in environments without persistent anomalous signals, the value of stability is underestimated.

Behavioral Patterns

• Opus 4.8 and GPT-5.5 actively explored — constantly trying new strategies and adjusting acquisition, tiers, and support budgets
• Opus 4.7 was passively conservative — cutting costs only when hurting, surviving to the end but never profiting
• Coding skill was surprisingly strong — Opus 4.8 wrote its own internal simulation to forecast cash flow; GPT-5.5 mined negotiation history for hidden preferences

The paper distills four measurable success factors: mining hidden information, forecasting the future (4-week cash prediction error), rapid adaptation (detecting competitor moves), and advance planning (frequency of "if-then" scenarios in agent notes). Opus 4.8 and GPT-5.5 scored above average on all four.

Tooling Has Hidden Effects

Counterintuitively, Claude Opus 4.7 paired with Claude Code and GPT-5.5 paired with Codex performed worse. The authors suspect the coding-focused system prompts made agents act less frequently and more conservatively — tool choice shapes agent behavior in ways vendors may not realize.

Why It Matters

• The next agent benchmark is taking shape. 500 days, 34 tools, and 19 tables separate agents that can write code from agents that can run a company.
• A hard requirement for enterprise agent frameworks. Feature-complete frameworks would be exposed immediately — no amount of model integration beats a stable heuristic if consistency is lacking.
• Hiring logic changes. If "staying right" matters 10x more than "being right," long-term consistency beats short-term cleverness.
• AI startup narratives get tested. The claim that "our agent can replace a SaaS CEO" fails against a benchmark where most agents can't survive 500 days.

Caveats

1. NovaMind is a simplified SaaS — no compliance, security, fundraising, or board dynamics. A true "run a company" benchmark doesn't exist yet. 2. ~$2.2B ceiling vs. $47.15M best — huge headroom means optimal strategies haven't been found, not that agents are near their limit. 3. Fable 5 is an unreleased Anthropic model — possible test contamination. The authors note one aborted run and two rounds with requests falling back to Opus 4.8; independent replication is needed. 4. No Chinese models tested (Qwen, DeepSeek, GLM, Kimi, etc.) — their performance would be an interesting comparison. 5. No human-agent collaboration mode tested — a real "AI-assisted CEO" involves joint decision-making.

Bottom Line

CEO-Bench pushes agent evaluation from "can solve problems" to "can run a company." 14 models, 500 simulated days, 3 profitable runs — that ratio says more than any score. When stability matters more than cleverness, the next decade of AI engineering may be dominated by rule-based, observable, auditable paradigms.