Paper Overview
Field: ML Authors: Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave, Chris Cundy Published: 2026-10-08 arXiv: 2610.12445
Abstract (translated)
Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up.
To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as *introspective deception*, the ground truth can only be determined through careful elicitation or by inspecting the model's training data. In one such evaluation, our probes distinguish transcripts containing the model's true hidden goal from other goals with up to 99.7% AUC.
Our probes also easily detect lying on politically sensitive topics by well-known open-weight models, as well as models concealing their own beliefs under pressure. We open-source our training dataset, FIBS, to enable effective deployment of probes at the frontier and encourage the community to further expand deception and sabotage cases.
Key Takeaways
- White-box probe-based deception detection scales to frontier monitoring settings
- Novel architecture aggregates signals across many layers and tokens
- 98.8% AUC on SHADE-Arena, beating a strong Opus 5.5 text-monitoring baseline
- Up to 99.7% AUC detecting hidden objectives in "introspective deception" cases
- Detects open-weight models lying on politically sensitive topics and models hiding their own beliefs under pressure
- FIBS dataset open-sourced for further research