Screen Before You Serve: Dying Ten Thousand Times in a Crystal Ball — Nubank's AI Customer-Service Evolution
> Analysis of the paper: *Screen Before You Serve: Production Learnings from Large-Scale LLM Agent Simulation at Nubank* (arXiv:2609.30137)
The core idea
Aviation never lets pilots make their first fatal mistake with passengers aboard — every deadly error is made first in a flight simulator. Nubank applies the same philosophy to its AI customer-support agents. Rather than exposing real bank customers to an unproven agent, engineers first let candidate versions "die ten thousand times" inside a simulator before any online A/B test.
Why simulation is necessary in banking
- Nubank's Card Delivery and Card Management agents perform real account actions: blocking lost cards, reissuing, updating addresses, verifying identity — via tool calls touching real accounts.
- Error tolerance is minimal: wrong account info is a data leak; skipping identity checks is fraud exposure; failed handoffs can become lawsuits.
- Handwritten test cases only cover what engineers imagine; online A/B tests risk customer trust. Simulation offers a third path: on-policy tool-boundary simulation.
- Inputs: an agent description, tool definitions, optional simulation prompts and historical conversations. An internal pipeline infers use cases, personas, tool relationships, and consistency rules (the "agent profile").
- An orchestrator distributes conversation quotas across use-case × interaction-style slices.
- Tool-boundary mocking: agent tool calls never reach production; a persona agent synthesizes schema-consistent responses, so the same record queried twice returns identical data. No shared database exists — every conversation runs in its own sandbox, enabling massive parallelism (100 trajectories in <10 minutes).
- P1 (length stats): simulated users are more verbose than real ones (median 111 vs 19 words per turn; median 6 vs 3 turns) — an honest, known limitation.
- P2 (embedding distance): Snowglobe conversations are closer to production than to an off-topic control (minimum cosine distance 0.035).
- P3 (score correlation): Snowglobe preserved production ranking extremes — best version ranked first in 96.69% of bootstrap resamples, worst ranked last in 100%. A naive LLM simulator inverted the ranking (best version ranked first 0% of the time). Pearson r = 0.74, Kendall τ = 0.67.
- P4 (human blind test): 7 domain experts misidentified 15 of 50 simulated conversations as real.
- Iteration speed: average version cycle dropped from 21.2 days (Card Delivery, 10 versions / 212 days) to 4.4 days (Card Management, 5 versions / 22 days) — a 4.8× speedup.
- Quality: unnecessary-human-handoff failure rate fell from 22.4% to 16.0%; Card Management vs Card Delivery: SSR +4.90 points (95% CI [4.08, 5.71]), tNPS +36.69 (95% CI [32.81, 40.57]).
- Model migration: 29 open-weight configs (GPT-OSS-120B, Nemotron-3-Super-120B-A12B, Qwen3.5-122B-A10B, Qwen3.6-35B-A3B) evaluated across 16,000+ simulated conversations. Qwen3.5-122B-A10B (high reasoning) was selected despite a status-retrieval weakness (35.2% vs incumbent GPT-5.2's 2.4%), which was addressed with targeted prompt optimization first. Live A/B: SSR +8.82 points (95% CI [7.95, 9.69]) — Nubank's historical best; tNPS −1.21 (not significant); p95 latency −25%.
- Rossell, K., et al. (2026). *Screen Before You Serve: Production Learnings from Large-Scale LLM Agent Simulation at Nubank*. arXiv:2609.30137. https://arxiv.org/abs/2609.30137
- Gupta et al. (2026). *Building customer support agents at 100M-user scale: an evaluation-driven framework*. KDD 2026 Industrial Track.
- Guardrails AI (2025). *SnowGlobe: The Simulation Engine for AI Agents and Chatbots*.
How Snowglobe works
Validating the mirror (P1–P4)
Hypothesis-driven screening pipeline
A fixed simulator configuration provides a shared baseline; every candidate change is scored against it with a pre-registered, falsifiable hypothesis and a pre-defined acceptance bar before entering online A/B. New evaluators must also score the baseline trajectories to detect evaluator bias. The pipeline even caught a staging service missing its tool-call parser.
Production results (all from online A/B tests)
Honest limitations
Simulation deliberately stops at the tool boundary: backend behavior, real latency, persistent state, and side effects are all out of scope. Simulated trajectories must be schema-aligned with production or export/reconciliation costs eat the gains. Verbose simulated users skew the stress distribution. As the paper puts it: the simulator's value is in supporting development decisions, not perfectly replicating production.
Takeaways for practitioners
1. Mock at the tool boundary, not via a shared database — independent sandboxes give parallelism and zero contamination. 2. Validate simulators in layers: length stats, embedding distance, score correlation, human blind tests. 3. New evaluators must first score the baseline, or you're measuring the ruler, not the agent. 4. Write hypotheses and acceptance criteria before running simulations. 5. Simulators catch engineering failures (e.g., missing tool-call parsers), not just intelligence gaps.
Conclusion
Trust in AI agents should rest not on a model's brand but on a layered, auditable verification process: simulation absorbs large-scale failure cheaply, online A/B validates survivors in a small blast radius, and production monitoring watches continuously. Nubank's paper is aviation's and pharma's century-old testing wisdom, transplanted onto AI agents.