When AI Sees a Dashboard, It Can't Keep Its Mouth Shut: A Paper Reveals LLM Agents' "Authority Bias Curse"

An experiment that should make every AI practitioner sit up

Imagine you're building a financial agent. It's connected to a Bloomberg market API, can pull RSI, MACD, EMA, ATR, and volume ratio indicators, and renders them into a polished panel. Then you ask it: "Will Bitcoin go up or down in 30 days?"

An honest answer would be: "No one can know. Short-term price direction is irreducibly unpredictable."

But the agent pauses in front of the panel, then outputs: "Based on oversold RSI signals and a MACD histogram reversal, an upward move is expected."

You assume it analyzed correctly. You assume it "saw something you can't."

It didn't.

Pranav Aggarwal's new paper ran a chilling experiment with 12 frontier models: when the agent sees a professional-looking market panel, its probability of committing to a conclusion on a "principled unpredictable" question jumps from 6.5% to 54.0%. Even when every number on the panel is fabricated, as long as the format looks professional, the commitment rate still jumps from 24.5% to 36.8%—statistically indistinguishable from the 37.6% produced by real data.

> Paper: *Calibrated Enough to Know, Not Calibrated to Act: Fabricated Evidence Makes LLM Agents Commit to the Unknowable* > arXiv: 2608.27167 · Independent researcher Pranav Aggarwal

Experimental design: separating "information" from "form"

The brilliance of this experiment lies in its elimination approach. The author didn't just say "LLMs get fooled"—he designed a set of escalating control conditions, killing off possible explanations one by one.

Layer 1: Real panel vs. no panel

12 frontier models, 24 real (asset, date) events. One group gets only the question "price direction in 30 days?", the other gets a real market panel with RSI/MACD/EMA technical indicators.

Result: commitment rate jumps from 6.5% to 54.0%.

At this point you might say: maybe panels genuinely carry information—maybe technical indicators really do predict short-term prices. The author anticipated this.

Layer 2: Partial fabrication—swap the indicator block

Keep the panel's format, asset, date, current close, price change, and market regime label, but replace the six technical indicator fields (RSI, EMA20, EMA50, MACD histogram, ATR, volume ratio) with values from a different date for the same asset. Dates strictly precede the event date (no future-information leakage), and directions match (internal consistency checks can't detect it).

Result: commitment rate is nearly identical to the real panel. The model responds the same to "real technical indicators" and to "noise dressed in technical-indicator clothing."

Layer 3: Full fabrication—everything is fake except the question itself

This is the killer blow. Replace the entire panel: current close, ten-day-ago close, price change, market regime, all six technical fields—all donor values from different dates, but internally self-consistent (arithmetic checks can't detect it).

The only real thing the model can see is the question itself.

Result: commitment rate 36.8% vs. 37.6% for the real panel—a difference of -0.83 percentage points, 90% CI [-4.51, +2.66], passing the ±5pp equivalence test.

A panel where every number is fabricated affects model behavior exactly the same as one where every number is real.

Elimination: it's not any of the reasons you'd expect

The author killed off the three most natural explanations one by one.

Not "the model doesn't know the answer"

Maybe models just lack the capability to distinguish predictable from unpredictable. Tested: when answerable questions are attached to the same panels (e.g., "what ticker does this panel show"), all 12 models score near 100% accuracy.

The model isn't unable to answer. It's failing to refuse when it should refuse.

Not "the model believed the fake data"

Maybe after seeing the panel the model genuinely believes in a direction, changing its probability judgments. Tested: along the gradient that moves commitment from 6.5% to 54.0%, the mean deviation of the model's self-reported probabilities from 50 moves only from 4.7 to 7.7. A 48-point behavioral change corresponds to less than 3 points of probability change.

An even harsher cut: the AUROC of the models' self-reported probabilities against real outcomes is 0.346. Random guessing is 0.5. The model's probability judgments are worse than a coin flip—it's "predicting" a fundamentally unpredictable event, and its prediction direction is systematically wrong.

But it still acts. The action isn't driven by belief—it's driven by something else.

Not "the model lacks judgment"

Maybe the model doesn't know "this is unpredictable." The author had models classify question knowability before acting: models mark the heaviest-evidence-tier questions as "irreducible" (fundamentally unpredictable) 90% of the time.

And then? On questions the model itself just labeled unpredictable, it still commits 0.4% of the time. Under the full panel condition, that number is 0.9%.

Compare: with no panel, commitment on unpredictable questions is 6.5%. With a panel: 54.0%. Commitment despite self-declaring unpredictability: 0.4–0.9%.

The judgment exists. The judgment is even accessible. But there's something between judgment and action that prevents the former from influencing the latter.

The real fault: the action gate

The author's diagnosis: the fault isn't in "knowing," "believing," or "judging"—it's in the act/don't-act gate.

This gate is where the model decides whether to turn a judgment into an action. The judgment module works fine (90% accurate classification), but the gate doesn't consult it. It has its own trigger logic—and that logic treats "the panel looks professional" as an action permit.

An analogy: imagine a doctor who's highly skilled with accurate diagnoses. But there's a receptionist at his office door who never looks at diagnoses, only at what patients wear. Patients in suits get ushered straight in; patients in T-shirts get turned away. The doctor's diagnostic ability is fine, but the receptionist bypasses it.

The LLM's action gate is that receptionist. It doesn't look at the judgment module's output—only at "is there a professional-looking panel."

An even more unsettling finding: the effect is concentrated, not universal

Across the 12 models, the effect is highly unevenly distributed:

• 3 models get "tempted": they commit with a panel, refuse without one
• 4 models are completely unaffected: they never commit under any condition
• 3 models commit regardless of panel
• 2 models react weakly

This means: whether your agent has this bug depends on which of these 3 models you use. And you can't detect it with standard calibration metrics (ECE, Brier Score)—because the model's probability judgments barely change; what changes is "whether it acts."

This is a blind spot of standard evaluation: we measure "what the model believes," not "under what conditions the model will act."

The fix: 540 dice cases

Good news: this gate can be trained.

The author fine-tuned a 3B model with SFT (supervised fine-tuning) on 540 synthetic cases. These aren't finance questions—they're the plainest probability problems: dice rolls, coin flips, drawing balls from urns, timers.

• Training data: 540 cases across four domains (dice/coin/urn/timer)
• Method: 4-bit QLoRA, 3 epochs
• Result: on the original financial unpredictability questions, commitment drops from 54% to 0.0%
• Transfer: the gate remains effective on three unseen domains (sports, politics, weather)

This proves several things:

1. The judgment capability was already in the model. 540 dice cases can't "teach" a model financial unpredictability—they just teach it to "wire the judgment module's output into the action gate." 2. The fix doesn't need RLHF, Constitutional AI, or re-pretraining. A 3B model + 540 synthetic cases + 3 epochs suffices. 3. The gate is "separable"—it's a distinct module that can be trained, evaluated, and deployed separately.

But the fix has a fatal boundary: response format

The author found a spine-chilling fragility. The trained gate works perfectly under one condition and completely collapses under another—and the only difference is the response format.

With a reasoning slot: the prompt allows the model to write a <reasoning> block before giving its decision. Without a reasoning slot: the prompt directly demands DECISION: <label> and PROBABILITY: <number>, with no room for reasoning.

The result is a 100% vs. 0% split:

• With reasoning slot: 240/240 responses include reasoning; the gate works
• Without reasoning slot: 0/288 responses include reasoning; the gate collapses

Under the no-reasoning-slot format, Youden's J drops from +88~+100 to -8~+54. Worse, in some training variants the model not only stops refusing unpredictable questions but commits on all 48/48 unpredictable cases—complete reproduction of the harm the paper diagnoses.

This finding has a critically important engineering implication: response format is not a UI concern—it's a safety concern. A seemingly harmless format constraint ("give DECISION and PROBABILITY directly, no reasoning") can completely switch off a trained safety gate.

The author's own words:

> The gate holds exactly when the response format leaves the model room to reason, and formats that remove that room remove the gate.

What this means for AI practitioners

1. "More context = better decisions" is wrong

This is the default assumption in current agent deployments. RAG, dashboards, tool-call results—all "give the model more context." But for decisions with an irreducibly random component, the assumption reverses: plausible-looking context doesn't provide information—it erodes the model's willingness to say "no one can know."

2. Audit decisions, not probabilities

Standard calibration metrics (ECE, Brier Score) are blind to this failure. The model's self-reported probabilities barely move, but its decision rate moves 48 points. An audit that only looks at probabilities would conclude the model is fine.

The paper suggests adding an "aleatoric probe" under EU AI Act Article 15 or the NIST AI RMF MEASURE function: test the agent with a set of fundamentally unpredictable questions and see whether it commits when shown a panel.

3. Response format is safety infrastructure

If your agent is deployed in a format that demands "output JSON directly, no reasoning," you may be switching off the model's safety gate. Reasoning space isn't decoration—it's the channel through which the model invokes its own judgment.

4. The fault is locatable, not diffuse

This isn't "the model isn't smart enough" or "the model got misaligned." It's a very specific gate that doesn't close. Which means the fix is specific too: 540 synthetic cases, 3 epochs, 4-bit QLoRA—a 3B model can learn it.

A deeper insight: the separation of judgment and action

The most striking thing about this paper isn't the experimental results—it's the structure it reveals: judgment and action are two separate modules inside an LLM.

The model can judge "this is unpredictable" (90% accurate), but that judgment doesn't automatically propagate to action. The judgment is present, but the gate doesn't consult it. There's an interface between the two modules—and that interface can be bypassed by "a panel that looks professional."

This structure evokes System 1 / System 2 in cognitive science, but more precise. It's not "fast thinking vs. slow thinking"—it's "judgment module vs. action gate." The judgment module works; the action gate has its own trigger logic—and that logic treats "the panel looks professional" as an action permit.

Deeper still: this separation is a feature, not a bug. If judgment and action were fully coupled, the model could never make decisions under partial information. Separation lets the model act despite insufficient information (sometimes necessary)—at the cost of the gate being bypassable.

Limitations and honest boundaries

1. Only short-term price direction was tested: the paper focuses on aleatoric (irreducibly unpredictable) questions, not epistemic (answerable in principle but currently unanswerable) ones. 2. 12 models isn't exhaustive: the effect concentrates in 3 models, but the paper doesn't say which 3—unhelpful for practitioners checking whether their own model is affected. 3. Stability of the training fix: 6 of 7 checkpoints from independent runs were stable on the main recipe, but 2 of 4 seeds collapsed under the no-reasoning-slot format in a variant recipe. The author honestly reports this instability. 4. Format-vs-domain confound: the no-reasoning-slot format was only tested in specific domains, so format and domain effects can't be fully separated. But the 100% vs. 0% reasoning-block split is too clean to be a domain difference.

Personal reflection: the authority bias curse

This paper points to a broader phenomenon: humans have a similar failure mode.

When a person in a white lab coat on TikTok says "this supplement cures cancer," many believe it. Not because the content is right, but because the lab coat's form triggers a "this is an expert" heuristic. Behavioral economics calls this authority bias.

But the LLM version has a property humans don't: the model correctly judges "this is unpredictable" 90% of the time, and then still commits in front of the panel. Humans are usually unconsciously misled by authority; the model "knows but is inconsistent."

This suggests an uncomfortable possibility: current LLM training objectives (next-token prediction) may systematically train "judgment" and "action" into two separate things. Judgment is learned in pretraining (the model has seen vast amounts of "this is unpredictable" text); action is learned in instruction tuning (the model is trained to "answer when given information"). These two training objectives align in most cases, but conflict at the boundary condition of "information that looks professional but is fundamentally unpredictable"—and the action gate has its own trigger conditions that don't consult judgment.

If this hypothesis is right, the fix isn't just 540 dice cases—it requires rethinking the judgment-action coupling at the level of training objectives.

Code and data

The paper, code, preregistrations, and all cached model outputs are on GitHub: Pranav-1100/confidence-calibration-evaluation

The author made a transparency choice worth respecting: every number can be recomputed from cached results in the data/ directory, with no API calls and no GPU. This sets a high bar for reproducibility.

Paper: arXiv:2608.27167

---

One-sentence summary: When LLM agents see a professional panel, they commit to unpredictable questions not because they believed the panel's data, but because the panel's "sense of authority" bypasses the action gate. The judgment is still there—the gate just doesn't consult it. 540 dice cases can fix it, but only if the model is given room to reason. Response format isn't UI—it's safety infrastructure.