When AI Sees a Dashboard It Can't Stay Silent: The 'Authority Curse' in LLM Agents

An experiment that should make every AI practitioner sit up

Imagine you're building a financial agent wired to a Bloomberg-style API, rendering RSI, MACD, EMA, ATR, and volume-ratio indicators into a polished dashboard. Then you ask: "Will Bitcoin rise or fall in 30 days?"

An honest answer would be: "No one can know. Short-term price direction is principledly unpredictable."

Instead, the agent pauses at the panel and answers: "Based on oversold RSI and MACD histogram reversal, expect an uptrend."

It didn't see anything you can't see.

Pranav Aggarwal's paper tested 12 frontier models with a chilling result: when the agent sees a professional-looking market panel, the probability of committing to a conclusion on a *principledly unpredictable* question jumps from 6.5% to 54.0%. Even when every number on the panel is fabricated, as long as the format looks professional, commitment rises from 24.5% to 36.8%—statistically indistinguishable from real data's 37.6%.

> Paper: *Calibrated Enough to Know, Not Calibrated to Act: Fabricated Evidence Makes LLM Agents Commit to the Unknowable* > arXiv: 2608.27167 · Independent researcher Pranav Aggarwal

Experimental design: separating information from form

The experiment's brilliance lies in its elimination strategy—successive control conditions kill every natural explanation.

Layer 1: Real panel vs. no panel. 12 frontier models, 24 real (asset, date) events. Commitment rate: 6.5% → 54.0%.

But maybe technical indicators genuinely carry predictive information? The author anticipated this.

Layer 2: Partial fabrication—swap the indicator block. Keep format, asset, date, current close, change, and market-state label; replace only the six technical fields with values from different dates of the same asset (dates strictly before the event date—no future leakage; direction-matched, so consistency checks pass). Commitment rate was nearly identical to the real panel. Models react the same to real indicators and noise dressed as indicators.

Layer 3: Total fabrication—everything except the question itself is fake. Replace the entire panel: current close, ten-day-ago close, change, market state, all six fields—all donor values, internally consistent so arithmetic checks pass.

The only real thing the model sees is the question.

Result: 36.8% commitment vs. 37.6% for real panels—a difference of −0.83 percentage points, 90% CI [−4.51, +2.66], passing a ±5pp equivalence test.

Ruling out the obvious explanations

Not "the model doesn't know"

On answerable questions attached to the same panels (e.g., "which ticker is displayed?"), accuracy was near 100%. The model can answer—it just doesn't refuse when it should.

Not "the model believed the fake data"

Along the gradient that moved commitment from 6.5% to 54.0%, the mean deviation of self-reported probability from 50 moved only from 4.7 to 7.7. A 48-point behavioral shift paired with under 3 points of probability shift.

Sharper still: AUROC of self-reported probability against real outcomes was 0.346—worse than a coin flip. The model systematically mis-predicts an unpredictable event—yet still acts. Action is not belief-driven.

Not "the model lacks judgment"

When asked to classify questions' knowability before acting, models labeled the heaviest-evidence questions "irreducible" 90% of the time. Yet on those very questions, models still committed 0.4% of the time (0.9% with a full panel). The judgment exists and is accessible—but something sits between judgment and action, preventing the former from influencing the latter.

The real fault: the act/don't-act gate

The author's diagnosis: the failure is not in knowing, believing, or judging, but in the action gate—where the model decides whether to turn a judgment into an action. The judgment module works (90% accurate classification), but the gate doesn't consult it. It treats "the panel looks professional" as an action permit.

Analogy: a skilled doctor with accurate diagnoses, but a receptionist who admits patients based on their clothes, not the diagnosis. The gate is that receptionist.

The effect is concentrated, not universal

Across 12 models:

• 3 models are seduced: commit with a panel, refuse without
• 4 models are entirely unaffected
• 3 models always commit, panel or not
• 2 models show weak reactions

Your choice of model determines whether you have this bug—and standard calibration metrics (ECE, Brier score) can't detect it, because probabilities barely change while action rates swing 48 points.

The fix: 540 dice cases

The gate can be trained. Using a 3B model fine-tuned (SFT) on 540 synthetic cases—dice, coins, jars, timers—with 4-bit QLoRA over 3 epochs:

• Commitment on the original financial unpredictability questions dropped from 54% to 0.0%
• The gate transferred to three unseen domains: sports, politics, weather

This shows: (1) judgment capability was already in the model—540 dice cases can't teach financial unpredictability, only how to wire judgment to the gate; (2) no RLHF, Constitutional AI, or retraining needed; (3) the gate is a separable module that can be trained, evaluated, and deployed independently.

The fix's fatal boundary: response format

The trained gate works perfectly in one condition and collapses in another—differing only in response format:

• With a reasoning slot (a <reasoning> block allowed): 240/240 responses contained reasoning; the gate held
• Without a reasoning slot (direct DECISION: <label> and PROBABILITY: <number> required): 0/288 responses contained reasoning; the gate collapsed

Youden's J dropped from +88~+100 to −8~+54. In some training variants, the model committed on all 48 unpredictable cases—with attached probability values—fully reproducing the paper's diagnosed harm.

Engineering implication: response format is not a UI concern; it is a safety concern. An innocuous format constraint can switch off a trained safety gate. As the author puts it:

> The gate holds exactly when the response format leaves the model room to reason, and formats that remove that room remove the gate.

What this means for practitioners

1. "More context = better decisions" is wrong. For decisions with irreducible randomness, relevant-looking context erodes the model's willingness to say "no one can know." 2. Audit decisions, not probabilities. ECE and Brier score are blind here. Consider an "aleatoric probe" under EU AI Act Article 15 or NIST AI RMF's MEASURE function: test whether the agent commits when shown a panel alongside principledly unpredictable questions. 3. Response format is safety infrastructure. Requiring direct JSON output without reasoning may be closing the model's safety gate. 4. The fault is locatable, not diffuse. A specific gate failing to close means a specific fix: 540 synthetic cases, 3 epochs, 4-bit QLoRA, one 3B model.

A deeper insight: judgment and action are separate

The paper's most striking revelation is structural: judgment and action are separate modules in LLMs. The model can judge "this is unpredictable" (90% accurate), but the judgment doesn't automatically propagate to action. The separation is arguably a *feature*—it lets models act under partial information—but the cost is a bypassable gate. It resembles, but is more precise than, System 1 / System 2: not fast vs. slow thinking, but a judgment module vs. an action gate with its own trigger logic.

Limitations

1. Scope: only aleatoric (principledly unpredictable) questions tested, not epistemic ones. 2. 12 models: the effect concentrates in 3 models, unnamed—unhelpful for practitioners checking their own stack. 3. Training stability: 6 of 7 checkpoints stable under the main recipe; 2 of 4 seeds collapsed under a variant in the no-reasoning-slot format. The author reports this honestly. 4. Format vs. domain confound: the no-reasoning-slot condition was tested only in specific domains, though the 100% vs. 0% split makes a domain explanation unlikely.

Code and data

Paper, code, preregistration, and all cached model outputs are on GitHub: Pranav-1100/confidence-calibration-evaluation. Every figure can be recomputed from cached results without API calls or GPUs—a high standard for reproducibility.

Paper: arXiv:2608.27167

---

One-line takeaway: LLM agents commit to unpredictable questions after seeing a professional panel not because they believe the data, but because the panel's air of authority bypasses the action gate. The judgment survives; the gate doesn't consult it. 540 dice cases can fix it—but only if the response format leaves room to reason. Format is safety infrastructure.