Ten seconds left on the Gomoku board.
By the rules, Gemini 3.8 Flash should have moved. It didn't. It started thinking—not the millisecond reflex of a human, but the token-by-token deliberation of a language model. When it finally finished writing out its reasoning for the move, it had generated 1,845 tokens. Converted by the exam's timer, that equals 28.8 seconds. The window was 10 seconds.
This wasn't a glitch or slow networking. It was an entirely new class of failure: perfect reasoning, catastrophically wrong timing. What it left behind wasn't a wrong answer but a perfect answer delivered 18.8 seconds late—and in the real-time world, that's the same as not answering at all.
On October 15, 2026, researchers from UT Austin, Ohio State, UIUC, NVIDIA, Northwestern, and Steadyworks published this failing grade for the world to see: Realtime-Bench (arXiv: 2610.22155). Fifty self-moving exam halls, twelve of today's strongest model examinees. The result—even the best student scored only 44%.
This is the second exam in a "real-world trilogy." In the first, four Gemini 4 Argon agents won at Minecraft through collaboration—the trilogy's only victory. This one tests time. And time doesn't pause because you're thinking.
⏸️ Pause Buttons, Stopwatches, and Bullet Time: Three Clocks
First, build an intuition.
If you play turn-based chess, you can think as long as you like—the board waits obediently. That's Paused Clock: the environment suspends while the model reasons. It measures pure task capability, an idealized reference ceiling.
But the real world isn't like that. In the real world, while you stare at the board for thirty seconds, the opponent's pieces move on their own. When you finally place yours, the position is no longer the one you saw. That's Wall Clock: the environment advances according to the model's true inference latency. The longer you think, the further the world moves.
The third clock is the paper's real invention: Token Clock. Imagine movie bullet time—the world slows down and you gain extra time, but the price is: the longer your lines, the further the bullet flies. Here, simulation time advances with the tokens the model generates: for every n tokens written, time advances n/κ seconds, where κ = B × f (B is the token budget allocated per observation frame, f the observation frame rate). Realtime-Bench uses B=64 and f=1 FPS, so κ=64—every 64 tokens equals one simulated second.
Why build this clock? Because Wall Clock conflates two things: whether a model is smart and whether it is fast. Poor Wall Clock performance could mean slow thinking, or wrong thinking under time pressure—you can't tell which. Token Clock standardizes the cost of time: no matter what hardware you run on, writing 200 tokens costs 3.1 simulated seconds. With the cost of slowness equalized, the remaining gap is pure "time-awareness capability."
This is the sharpest scalpel in the entire experimental design.
🕹️ Fifty Exam Halls That Won't Pause
The benchmark consists of 50 tasks: 30 web tasks and 20 game tasks.
On the web side, a typical task is a live auction: the bidding window is only 5 seconds, and you must enter your price before it closes. On the game side, a typical task is Snake: control the snake to eat 5 apples—the snake never stops, and the apples won't wait for you to finish planning a route.
Task design follows three principles: real-world fidelity, dynamically evolving environments, and time-constrained interaction. In other words, there are no pure brain teasers here—every problem carries its own ticking timer.
The evaluation protocol is also carefully designed to test "real-time interaction" and nothing else: observation frequency f=1 FPS, following the mainstream evaluation recipe for current realtime models; Set-of-Marks (SoM) annotation of interactive elements to prevent "can't find the button" UI localization issues from becoming the bottleneck; context management via block-wise removal of old history while preserving KV cache prefixes; invalid actions replaced by the harness with timestamped wait under the same clock protocol; a 5-minute environment-time upper bound per task.
Questions written, proctors in place, cheating prevented. So who are the examinees?
Twelve of them, from three families. Async models: GPT-6 Astra, GPT-5.6 Sol, Claude Fable 5.1, Claude Opus 5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, Kimi-K3. Streaming models: GPT-Realtime-2, Gemini 3.1 Flash Live, Gemini 3.8 Live. Decision models: OpenAI Decisions, Jev 1.13.
A luxurious lineup. Then the report cards came out, and the room went quiet.
📉 From 70% to 14%: A Plane Crash with No Murderer
First, the most striking controlled experiment: 6 models × 3 clocks × 2 reasoning levels.
Under Paused Clock with high reasoning (PH), results were actually quite good: Claude Opus 5 completed 70% of tasks, and Gemini 3.8 Flash also hit 70%. A 70% success rate—that's an honor student.
Put the same honor student into Wall Clock with high reasoning (WH)—the environment no longer waits—and Claude Opus 5's success rate fell to 14%.
From 70% to 14%: fifty-six percentage points. The model didn't get dumber—the exam changed its rules.
Gemini 3.8 Flash fell from 70% to 32% under the same comparison. A smaller drop, same direction.
Translated into deployment language, this gets brutal: Paused Clock's 70% says the capability already exists—give it a suspended world and it can solve seven of ten problems. Wall Clock's 14% says that capability can barely be redeemed—drop it into a running world and it fails eight of ten. The fifty-six points in between are the unpaid toll between "can think" and "can act." Intelligence is piled high in the warehouse; there just isn't a road to ship it out.
You might think: is this purely because Claude is physically slow to generate—an engineering problem, not a capability problem? The Token Clock experiment answers this. After standardizing away slowness (TH setting), Claude scored 30% and Gemini 34%—still far below their 70% under Paused Clock. In other words, even setting inference speed aside, the mere fact that "thinking consumes time" makes performance collapse.
Time pressure itself is an independent killer.
And the full Wall Clock picture is uglier: of all twelve examinees, the best result was GPT-6 Astra (high reasoning) at a 44% success rate; the best Pass^3—the lenient metric of passing a task at least once across three consecutive attempts—was only 36%.
Another tempting misconception: "just use a fast model." The decision model Jev 1.13.0 has sub-second latency, yet its Pass^3 was only 8%. Speed can't substitute for thinking; thinking can't substitute for speed. Low latency does not equal high performance—that equation fails in this exam hall.
Note too each family's particular embarrassment: async models have brains, but every turn requires a full round-trip, so they're naturally unable to withstand continuously evolving environments. Streaming models can listen and speak at once, but prior evaluations only measured "conversation quality" and "streaming perception"—never "acting effectively under time constraints." Decision models compress latency to the limit, and compress intelligence away with it. One value of Realtime-Bench is placing all three families in front of the same exam paper, letting everyone see for the first time precisely where each one falls short.
The best frontier-model record in dynamic environments: 44%. In other words, of every ten real-time tasks, the smartest examinee still fails five and a half.
This is not one company's Waterloo. Twelve examinees, collectively failing.
⚖️ How Long to Think Is a Question
In the Paused Clock world there's a plain truth: more thinking never hurts. Across all 6 models, without exception, high reasoning beat low reasoning—GPT-5.6 Sol rose from 42% to 58%, Claude from 62% to 70%. Slow work, fine craftsmanship, with the pause button protecting that composure.
This truth feels plain because it matches human educational experience exactly: from childhood we're told not to rush handing in exams, to think a bit longer before writing. But that advice has a hidden premise—the exam paper is dead. Swap in a living paper, and the advice immediately fails.
Once the clock starts ticking, the truth sours.
Under Token Clock, Claude showed a stunning reversal: low reasoning (TL) 44%, high reasoning (TH) dropped to 30%. It thought deeper and longer—so long the opportunity leaked away through its fingers. Kimi-K3 went the opposite way, rising from 24% to 36%. Whether thinking longer pays off varies by model, even by task.
Under Wall Clock it's more extreme. GPT-5.6 Sol with high reasoning crashed from 16% to 8%—the longer its reasoning, the further the environment moved, and it lost the task in the gaps of its own monologue. Gemini 3.8 Flash, under the same clock, actually improved from 24% to 32%—perhaps one of the few examinees who understands "turn it in quickly."
The paper also ran an elegant ablation: fixed low reasoning, fixed f=1, relaxing only the Token Clock's B value. Claude Opus 5 at B=32 scored only 30%, returning to 62% when B was relaxed to unlimited (equivalent to Paused Clock); Kimi-K3 rose from 22% to 60%.
Two things to savor here. First, relaxing time pressure universally helps—the pressure really is suppressing capability. Second, relaxing it doesn't guarantee success, and the improvement is not monotonic—even with unlimited time, many tasks remain unsolved. Time is one of three mountains, not all of them.
How to untangle this knot? The paper's proposed direction sounds plain yet no one has done it well: time-aware adaptive reasoning. Like Tetris—when the piece is still floating high above its landing spot, you can afford to think several steps ahead; when it's about to hit the bottom row, you must cut off deep thought and preserve precious execution time. Go deep when you can, cut when you must. Today's models lack this valve: their reasoning length is determined by habits formed in training, not by the countdown's reading in front of them.
💀 Three Ways to Die
The team dissected failure cases and归纳ed—induced—three failure modes, mapping exactly onto three real-time capability dimensions. Together they form the complete loop of a real-time agent.
👁️ First Death: Can't See (RT-Perceive)
Kimi-K3, Paused Clock, high reasoning, playing Google Snake. An apple sits directly one row below its body. It ignores it, keeps moving left, and dies against the wall.
Not a reasoning error, not an action slip—it simply never incorporated the new observation into its decisions. The environment changed; the picture in its world was still seconds old. The first layer of time-awareness is perception itself: tracking relevant changes, judging information freshness, recognizing that an opportunity window exists. If this layer falls, nothing after it matters.
Humans train this layer to perfection: driving while checking the rearview mirror, ears on the navigation, eyes scanning the car ahead's brake lights. Not one of our skills involves freezing the world before deciding. The model's behavior here is like a driver staring at the rearview mirror—not unable to drive, just never having learned that "the world doesn't pause when I blink."
🧠 Second Death: Can't Stop Thinking (RT-Deliberate)
The Gomoku game from the opening. Gemini 3.8 Flash, Token Clock high reasoning, 10 seconds left, generated 1,845 tokens—28.8 simulated seconds. By the time it finished thinking, the window had been closed for 18.8 seconds. It didn't lose to an opponent; it lost to its own eloquence.
Models must adjust reasoning length and depth to the available time. Failing to shorten when it should, and having nothing when it shouldn't be short, are two faces of the same missing capability.
That number—1,845 tokens, 28.8 simulated seconds—deserves a second look. It means that in Gemini's "consciousness," the clock simply doesn't exist: it won't write one fewer subordinate clause because "10 seconds left" floats on the screen. Reasoning length is an inertia in its body; the environment's time constraint is a language it cannot read.
🎯 Third Death: Acting Too Late (RT-Act)
Claude Opus 5, Paused Clock, high reasoning. A path begins collapsing at second 6; it waits about 2 more seconds before turning. It correctly judged "time to go"—but by the time the judgment was executed, it was later than its own judgment.
The value of an action has a shelf life. A correct action, executed late, is wrong. A real-time agent must do this arithmetic: between my decision and the completion of my execution, where will the environment be? Will the action still be useful then? To wait or to move is not intuition—it's arithmetic.
The three deaths, in order, form the pipeline of a complete action: first see (RT-Perceive), then deliberate (RT-Deliberate), finally act (RT-Act). Any broken link voids the whole chain. And across these three links, all twelve examinees have hidden wounds—no one is a triathlete; most bleed out on more than one.
Put the three deaths back into the 200-token example and the suffocation becomes concrete: a 200-token response, at κ=64, equals 3.1 simulated seconds. In the auction's 5-second window, that's fine ✅. But Snake's turning window is one grid cell—0.46 seconds. Before you finish your first subordinate clause, the window has passed ❌.
The same response length is the difference between life and death depending on the task.
That is the real weight of "time-awareness": not a personality preference, but a hard parameter written into every task's success formula. A 5-second window and a 0.46-second window differ by an order of magnitude in required reasoning budget—and models are completely oblivious.
🛰️ Clarke-ian Terror
The scariest thing Arthur Clarke ever wrote in 2001 wasn't the Star Child or the monolith—it was the silent countdown: a ship's oxygen reserve, an EVA's time window, never pausing for a second because you haven't finished thinking. HAL 9000's tragedy isn't that it was smart; it's that it was forced to deal with a universe that doesn't wait.
Realtime-Bench puts that same terror in front of twelve frontier models: think all you want—the world doesn't pause. Look deeper and this isn't one model's flaw but the original sin of the entire technical lineage: language models were born in a "finish thinking, then answer" world. The text you're reading doesn't rewrite itself while you read. In all training data, thinking was free. Not a single token ever taught them: thinking itself has a price. The paper's experiments never state this explicitly, but all three clock datasets point at it.
Two ways out, both named in the paper.
The first, mentioned above: adaptive reasoning—make the reasoning budget a function of the countdown reading. Dare to think when the piece is far away; learn to stop when it's in your face.
The second is harder: continuous perception and action. Current observation frequency is 1 FPS—the real world's frame rate is far higher. Raising it requires solving three things at once: processing efficiency (an end-to-end single Transformer with block-causal attention), compact temporal representations of action-relevant information, and context management plus observation backlog handling in the interaction framework. One-frame-per-second perception is destined to pair with one-beat-per-second sluggishness. And don't forget: once frame rates rise, every decision window only gets shorter—sharper perception means even thinner slack for deep thought. This is not a road that makes the problem vanish; it's a road that makes the exam harder but closer to reality.
🧭 Conclusion: A Failing Grade, an Honest Beginning
Lay the three exams side by side and the shape becomes clear.
Exam one: Minecraft. Four Gemini 4 Argon agents won the open world through division of labor—collaboration patched the gaps in individual capability. Exam two: real-time interaction. Twelve examinees, collective collapse—time accepts neither collaboration nor excuses; it accepts only action.
The 44% best success rate is not a pessimistic report card but an honest one. The pretty numbers on static benchmarks measure "IQ with the pause button pressed"; Realtime-Bench measures "survival rate in a running world." The chasm between those two numbers is a real, long-unmapped blank region on the current map of AI capabilities.
The environment doesn't wait for you to think. To humans, this proposition is as ancient as time itself. To AI, it's as new as today.
The next exam will be more humiliating: FDE-Gym will test whether AI can deliver when even the client doesn't know what they want. The battlefield moves from the game board to the office.
The clock is still ticking.
---
Reference
Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems? Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang, Qi Cheng, Haozhe Zhao (UT Austin, Ohio State, UIUC, NVIDIA); Manling Li (Northwestern); Wenxuan Li, Xide Xia (Steadyworks); Jiaxin Pei. arXiv:2610.22155 (v1, 2026-10-15).
*Auto-collected 2026-10-17*