Argo-Bench: The Best Models Only Reach 34.8% — Enterprise Data Work Isn't Just Writing SQL
Field: Agent Benchmark / Enterprise Data Authors: Joseph G. L. et al. Institution: TextQL Labs arXiv: 2610.02122 Code: github.com/TextQLLabs/Argo-Bench
A "God's-Eye View" of a Delivery Platform
Imagine you're a data analyst at a food delivery platform. The platform handles 220,000 orders a day — 81 million a year. You have 235 tables and 7.5 billion rows of data covering orders, couriers, merchants, payments, fraud, and customer service. Your ERP system closely resembles Oracle E-Business Suite, that living fossil of enterprise software.
Now your boss gives you a task: "Check whether any couriers in Brooklyn were over-penalized in March, and if so, refund them."
How would you do it? You wouldn't just write one SQL query. First you'd figure out which table stores "penalties," how "couriers" and "orders" are joined, how "Brooklyn" appears in the address fields, and what "over-penalized" actually means. Only then could you query. And after querying, you'd still need to execute the refund — not by editing a number in the database, but by going through the platform's refund process.
This is what Argo-Bench is designed to measure. It doesn't test "can you write SQL" — it tests "can you work like a real enterprise data analyst."
Why Text-to-SQL Isn't Enough
Existing text-to-SQL benchmarks (Spider, BIRD, etc.) test "given a question, write one SQL query." Real enterprise data work goes far beyond that:
1. Real-world scale: The largest Spider database has a few dozen tables and tens of thousands of rows. Argo-Bench's delivery platform has 235 tables and 7.5 billion rows. This is a qualitative shift, not a quantitative one — finding patterns across 7.5 billion rows requires understanding the data structure first, not scanning tables.
2. Actions are required: The endpoint of text-to-SQL is "return query results." The endpoint of Argo-Bench is "take action" — banning fraudulent accounts, allocating rider incentive budgets, issuing back pay. You must first establish the facts, then decide, then execute.
3. Information asymmetry: The agent sees the ERP warehouse, but the platform's "true state" is held by a simulator. Warehouse data can be stale or incomplete. The agent must "reconstruct facts" from the warehouse rather than directly read them.
How Argo-Bench Is Designed
The TextQL Labs team built a complete delivery-platform simulator:
- Scale: New York City, 2024, 81 million orders
- Data: 235 tables, 7.5 billion rows, based on an Oracle E-Business Suite schema
- Economic model: Realistic pricing, commissions, incentives, and fraud patterns
- Tasks: 210 data science and analytics tasks spanning 5 business domains
- The strongest model exceeded a score of 95 on only 34.8% of tasks
- The average score was just 59.5 out of 100
The five domains are: financial reconciliation, fraud detection, courier management, merchant operations, and market analysis. Every task requires the agent to reconstruct facts from the warehouse and then act.
The scoring is also interesting: it doesn't check whether your SQL is right, but looks at what consequences your actions produced in the simulator. Banning the correct fraudulent account? Points. Banning a legitimate user? Deductions. Doubling rider incentives? Deductions.
The Numbers
14 frontier and open-source models were tested. The results:
Where does the gap come from? From the two stages text-to-SQL doesn't cover: "reconstructing facts" and "taking action." Writing correct SQL doesn't mean you know which tables to look at, how to join them, or what to do once you have the results.
A Deeper Insight
Argo-Bench reveals an overlooked problem: the difficulty of enterprise data work lies not in "writing queries" but in "understanding the business."
This aligns with a broader trend in AI: evaluation is shifting from "testing skills" to "testing judgment." Spider tests SQL skill; Argo-Bench tests business judgment. The former can be replaced by tools (AI already writes SQL accurately); the latter requires a deep understanding of business logic, data structures, and causality.
One design choice deserves special attention: the simulator's true state is deliberately hidden from the agent. The agent can only see the ERP warehouse, which may be stale or incomplete. This simulates a core difficulty of real enterprises — you never see the complete facts; you can only reconstruct them from limited records.
This mirrors exactly what human analysts do. A senior analyst's value isn't knowing SQL — it's knowing what in the data is trustworthy, what is missing, and what needs cross-validation.
A Cross-Domain Analogy
Argo-Bench's design philosophy reminds me of flight simulators. A flight simulator doesn't just test whether you "can fly a plane" — it simulates real scenarios like engine failures, bad weather, and emergency landings, then checks whether you make the right decisions.
Argo-Bench does the same thing: it simulates a realistic enterprise data environment and checks whether an agent can make correct decisions with incomplete information. This ability to act under uncertainty is the core of enterprise data work.
An Honest Assessment
Argo-Bench's limitations are obvious. First, it simulates only one industry (delivery platforms), and data structures vary widely across industries. Second, 210 tasks is substantial, but still few compared to the variety of tasks a real enterprise data team handles daily. Finally, while the simulator's economic model is "based on real data," it remains simplified — real delivery platforms have far more complex strategic dynamics (couriers vs. platforms, merchants vs. platforms).
But as a first step from text-to-SQL toward real enterprise data work, Argo-Bench's contribution is clear: it demonstrates that the strongest models achieve only a 34.8% pass rate in realistic enterprise data scenarios, pointing the way for the next generation of agents.
---
Paper: arxiv.org/abs/2610.02122 Code: github.com/TextQLLabs/Argo-Bench