Paper Overview
Field: Machine Learning Authors: Erica Stutz, Giacomo Marino, Daniella Meeker Published: 2026-05-17 arXiv: 2505.12350
Abstract (English Translation)
Generative models are often trained with a next-token prediction objective, yet many downstream applications require the ability to estimate or control sequence-level properties. Next-token prediction can lead to overfitting of local patterns during training, underfitting of global structure, and requires significant downstream modifications or expensive sampling to guide or predict the global attributes of generated samples at inference time.
Here, we introduce Conditional Attribute Transformers, a novel method for jointly estimating the next-token probability and the value of an attribute conditional on each potential next-token selection. This framework enables three critical capabilities within a single forward pass, without modification of the input sequence:
1. Per-token credit assignment over the entire sequence, by identifying which tokens are associated with the attribute value; 2. Counterfactual analysis, by quantifying attribute differences based on alternative next-token selections; 3. Steerable generation, by decoding sequences based on a combination of next-token and attribute likelihoods.
Our method achieves state-of-the-art performance on sparse-reward tasks, improves next-token prediction at sufficient model scale, estimates attribute probabilities orders of magnitude faster than sampling, and can guide autoregressive sequence model decoding across a range of language tasks.
---
*Auto-collected on 2026-05-18*