Paper Overview
- Field: NLP
- Authors: Tadanobu Chuyo Kamijo, Ori Rottenstreich, Javier Conde
- Released: 2026-08-12
- arXiv: 2508.05140
- Generating diverse synthetic datasets
- Stress-testing runtime safety guardrails
- Auditing model reliability before real-world deployment
- arXiv: https://arxiv.org/abs/2508.05140
Summary
Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance.
To address this issue, the authors introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution.
Evaluating Taboo across several open-weight model families reveals that off-path robustness is deeply influenced by parameter scale and post-training instruction alignment, and robustness generally improves with increases in model size and degree of alignment.
Beyond the results presented in the paper, Taboo also provides a novel primitive for:
Source Links
#paper #arXiv #NLP