Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops

Forum topic · ✨步子哥 · 2026-10-14

Summary

A CMU paper by Yubo Li, Ramayya Krishnan, and Rema Padman introduces a diagnostic framework for System One models—fast LLM-based decision models that return probability distributions without chain-of-thought reasoning. By independently probing factual understanding and decision quality at each agent state, the authors build a 2x2 diagnosis matrix. Strikingly, for the six most accurate systems, 73-85% of decision errors occur on states where all fact probes were answered correctly, meaning models read the state right but acted wrong. Fourteen fast systems collectively failed on the exact same 141 states, revealing structural blind spots rather than random noise. The paper also identifies two distinct mechanisms—consolidation failure and capture failure—requiring different fixes. A practical self-summarization technique, where the model's own fact-probe answers are re-fed into the decision request, boosts accuracy by up to 27.6 points, while a confidence-gated cascade achieves 0.932 accuracy with only 38% of states needing a second pass.

The Problem That Keeps Agent Developers Up at Night

You built an LLM Agent. It runs fine in simulation, then suddenly picks a bizarre action at some state. You open the logs and find no chain of thought to inspect—because you're using a System One model, which returns only a probability distribution without any reasoning trace.

Now you face an unavoidable diagnostic fork:

Did the model fail to understand the current state? Or did it understand—but choose wrongly?

The former points to context representation, context budget, and state encoding; the latter points to the decision itself. If you can't distinguish these two failure types, your fix is a blind guess—more context? A stronger model? A new decision head?

CMU researchers Yubo Li, Ramayya Krishnan, and Rema Padman published a paper whose title *is* the question: "Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops". They don't just pose the problem—they build a ruler for it.

System One Models: The "Fast Thinking" of Agents

The dominant agent paradigm today: at every step, call an LLM, have it write a chain of thought, then output an action. That's System Two—slow, expensive, but inspectable.

A new class of models takes another path: System One models. Given an agent state and a set of candidate actions, a single forward pass returns a probability distribution—no text generated. Representatives include JEV (TypeSafe AI), OpenAI Decisions, Clef-27B (Cloudflare), and Eikos-27B.

How fast are they? The paper measured:

Median latency for a full state: 27–225 ms for System One models
Likelihood readouts from open-source LLMs: 317–947 ms
Reasoning models (GPT-6.1): 2.3 s

JEV processed 65,228 decision states for $8.79—about $0.13 per 1,000 states. GPT-6.1 would cost $642 for comparable coverage, 73x more.

But when a System One model errs, there's nothing in the logs. No reasoning, just a distribution. The error becomes a black box.

The Four-Quadrant Diagnostic: Opening the Black Box

The core method is elegant. At each agent decision state, ask two questions independently:

1. Decision question: which action to take next? 2. Fact probes: did the previous action succeed? Where is the agent? What is it holding? Is this action allowed?

Each probe is asked independently on the same state, without leaking answers to the decision question. Cross the two results into a 2x2 matrix:

| | Action correct (D=1) | Action wrong (D=0) | |---|---|---| | All facts correct (U=1) | U1D1: ideal | U1D0: read right, acted wrong | | Some facts wrong (U=0) | U0D1: lucky guess | U0D0: total failure |

The key quadrant is U1D0: all fact probes answered correctly, yet the action is wrong. That's "Reading Right, Acting Wrong."

The fork has clear engineering implications:

High U1D0 → the problem is in decision-making; upgrade the decision head or add reasoning
High U0D0 → the problem is in understanding; fix state representation or add context

Core Finding: 73–85% of Errors Are "Read Right, Acted Wrong"

Across 19,663 paired test states and 18 systems, the result is striking:

For the six most accurate systems, 73–85% of decision errors occur on states where all fact probes were answered correctly.

Moreover, this share grows with accuracy—stronger systems mainly reduce "reading wrong" errors, while "read right, acted wrong" barely shrinks. GPT-6.1, the reasoning reference, scores 0.884 accuracy, yet 85% of its errors are also in this quadrant.

This means: understanding and deciding are separable capabilities. You can have a model that perfectly understands the state while its decisions still collapse.

141 "Everyone Dies" States

If "read right, acted wrong" were each model's idiosyncratic noise, failures would spread uniformly across the state space.

They don't. Fourteen qualified fast systems all fail on exactly the same 141 states—all probes correct, action wrong. The null-hypothesis expectation is 0.

Each system scores 74–94% on states where only it fails, but only 1–16% on states where all other 13 systems fail.

Errors are not idiosyncratic—they attach to specific state properties. There exists a class of "structurally hard states" that expose a shared blind spot of all fast models.

Composition Failure: Can Read + Can Compute ≠ Can Compute While Reading

To determine whether U1D0 reflects "probes missed key facts" or "the model can't compose information," the paper constructs states with verifiable probe sufficiency:

4 rules + 1 default action, rule conditions determined by 8 binary facts
4 facts are rule-tested, 4 are distractors
Each fact assignment yields a unique gold action, verifiable by enumeration

The result is striking:

JEV and Qwen3.8-27B achieve near-perfect decision accuracy when given facts only (1.000 and 0.994), but only 0.791 and 0.797 when given the raw trace only.

They can read facts perfectly and derive actions from facts perfectly—but when both must happen together—deciding while reading a raw trace—accuracy drops by 20 points.

It's not "can't read" or "can't compute"—it's can't compute while reading. The paper calls this a composition failure.

Two Failure Mechanisms, One Symptom

Finer analysis reveals two distinct mechanisms hidden inside the U1D0 quadrant:

1. Consolidation failure — JEV, Qwen3.8-27B

Reads each fact correctly
Can derive actions from facts
But fails to consolidate facts scattered across the trace into a coherent snapshot before applying rules
Fix: gather facts into one place

2. Capture failure — Clef, Vega-27B, SimpleJev-27B

Reads each fact correctly
But gets captured by "true-but-irrelevant distractor cues" outside the procedure
Executes a plausible action not in the program, even when rule conditions are met
Fix: remove distractor facts (Clef's accuracy jumps from 0.589 to 0.924)

The same U1D0 label, two completely different mechanisms and two completely different repair paths. Final accuracy alone can never tell them apart.

Self-Summarization: Letting the Model Rescue Itself

The most practical finding: since System One models answer fact probes perfectly, what happens if they write their own summaries?

Procedure: 1. Ask the decision question on the raw trace 2. Ask fact probes in an independent request 3. Splice the model's own probe answers back into the trace, ask the decision again

Results:

| System | Baseline | + own answers | Gain | |---|---|---|---| | JEV | 0.791 | 0.968 | +17.7 | | Qwen3.8-27B | 0.797 | 0.955 | +15.8 | | Qwen3.5-9B | 0.564 | 0.767 | +20.3 | | SimpleJev-27B | 0.551 | 0.827 | +27.6 | | Clef | 0.589 | 0.747 | +15.4 |

The model reads the facts right, feeds them back to itself, and decisions improve. No external information, no stronger model—just decoupling "reading" and "deciding" into two requests.

A more radical option: compose with code. Since rules are explicit, have the model answer facts and map facts to actions in code. JEV reaches 0.996; Clef reaches 1.000.

This is the original design intent of System One models—many narrow questions in parallel, composed by code, echoing TypeSafe AI's design philosophy for models like JEV.

Cascading: Ask Twice Only When Uncertain

Two passes every time is wasteful. The paper tests a cascade:

First decision pass: if confidence > 0.7, use it directly
If < 0.7, ask fact probes, splice them back, decide again

JEV hits 0.932 accuracy with only 38% of states needing the second pass.

This is the most elegant engineering solution: use confidence as a gate and spend compute only on uncertain states. Decouple understanding and deciding—just not every time.

GPT-6.1's "Over-Caution"

An interesting finding: GPT-6.1 is the weakest of the six strongest systems on tau²-bench (customer-service policy scenarios, 0.709) while strongest elsewhere.

The cause isn't misreading—it's over-caution. When the gold action is "execute the write operation the user just confirmed," GPT-6.1 replies asking for confirmation instead of executing 16.7% of the time; fast systems do so only 2.8–7.0%.

The reasoning model's "thinking" teaches it to double-confirm, but the gold standard counts this as an error—revealing tension between reasoning models' caution and gold-standard efficiency assumptions.

Engineering Takeaways

1. Don't look only at accuracy. Two models with identical accuracy can have entirely different failure modes. Torchcast-27B reads best (U=0.858) but decides worst (D=0.678); SimpleJev-27B reads mediocre and decides worse—one is "read right, acted wrong," the other "captured by distractors."

2. Probes are diagnostic tools, not training targets. The authors explicitly call this "a diagnostic tool, not a leaderboard." Optimize probe accuracy as a training objective and the model will perform on probes without deciding better.

3. Cascades are the right way to use System One. Don't have one model do everything end-to-end; let narrow models answer narrow questions fast, compose with code, and escalate to slower paths only when uncertain.

4. "Understanding ≠ deciding" is a design principle. Your agent architecture should treat "reading the state" and "choosing the action" as independently replaceable components. When decisions go wrong, first ask "did it misread or mis-choose?"—then decide which layer to fix.

A Deeper Insight: Another Case of Scalar Illusion

The paper adds a powerful example to the "scalar illusion" lineage: accuracy is a scalar hiding two independent dimensions.

Model A at 0.78 accuracy may fail entirely by "misreading"
Model B at 0.78 may fail 80% by "reading right, acting wrong"

Their repair paths are completely different, yet accuracy says they're "equally good."

Like prior findings that benchmark accuracy ≠ representational faithfulness, the pattern repeats: when you measure a multi-dimensional system with a scalar, you systematically misjudge the repair direction.

Personal Reflection: Diagnostics vs. Leaderboards

What I admire most is the paper's stance. The authors explicitly say: "This is a diagnostic tool, not a leaderboard."

In an era of increasingly competitive AI evaluation, this attitude is rare. Most benchmarks compete on whose number is higher; this paper teaches you to understand why your model errs. It doesn't tell you which model is best—it tells you where your model fails and how to fix it.

Same logic as medicine: a thermometer doesn't cure, but it tells you which medicine to take.

Paper: https://arxiv.org/abs/2610.18579 HTML version: https://arxiv.org/html/2610.18579v1

---

*This paper offers every agent developer a practical diagnostic framework. Next time your agent picks a wrong action, don't rush to swap models—ask first: did it read the state right?*