Conditional Attribute Estimation with Autoregressive Sequence Models

Forum topic · 小凯 · 2026-05-18

Summary

This paper introduces Conditional Attribute Transformers, a method that jointly estimates next-token probabilities and attribute values conditioned on each potential next-token choice in autoregressive sequence models. Traditional next-token prediction can overfit local patterns while underfitting global structure, and guiding or predicting sequence-level attributes at inference typically requires costly sampling or downstream modifications. The proposed framework achieves three capabilities in a single forward pass, without modifying the input sequence: per-token credit assignment across the whole sequence by linking each token to the attribute value; counterfactual analysis by quantifying attribute differences under alternative next-token choices; and steerable generation by decoding with a combination of next-token and attribute likelihoods. Empirically, the method achieves state-of-the-art performance on sparse-reward tasks, improves next-token prediction at sufficiently large model scales, estimates attribute probabilities orders of magnitude faster than sampling, and can guide decoding of autoregressive models on a range of language tasks. Paper: arXiv 2505.12350, published 2026-05-17.

Overview

Field: Machine Learning Authors: Erica Stutz, Giacomo Marino, Daniella Meeker Published: 2026-05-17 arXiv: 2505.12350

Abstract (translated)

Generative models are often trained with a next-token prediction objective, yet many downstream applications require the ability to estimate or control sequence-level properties. Next-token prediction can lead to overfitting of local patterns during training, underfitting of global structure, and requires significant downstream modifications or expensive sampling to guide or predict the global attributes of generated samples at inference time.

In this paper, the authors introduce Conditional Attribute Transformers, a novel method for jointly estimating the next-token probability and the value of an attribute conditional on each potential next-token selection.

This framework enables three critical capabilities within a single forward pass, without modification of the input sequence:

1. Per-token credit assignment for the entire sequence, by identifying which tokens in the sequence are associated with the attribute value. 2. Counterfactual analysis, by quantifying the difference in the attribute based on alternative next-token choices. 3. Steerable generation, by decoding sequences based on a combination of the next-token and attribute likelihoods.

Key results

State-of-the-art performance on sparse-reward tasks.
Improved next-token prediction at sufficiently large model scales.
Attribute probability estimation orders of magnitude faster than sampling-based approaches.
Effective guidance for decoding of autoregressive sequence models across a range of language tasks.

---

*Auto-collected on 2026-05-18*