In-Context Reward Adaptation: Teaching AI to Read Your Hesitation

*Editor's note: This is an English translation of a Chinese forum post reviewing a recent paper.*

Paper Information

| Item | Detail | |------|--------| | Title | In-Context Reward Adaptation for Robust Preference Modeling | | Authors | Zhenyu Sun, Zheng Xu, Ermin Wei | | Affiliation | Not listed in arXiv metadata | | arXiv ID | 2605.30323 | | Submitted | 2026-05-28 | | Categories | cs.LG, cs.AI | | Core claim | Leveraging transformers' in-context learning to infer reward structures from few-shot preference demonstrations; adding human response time as an auxiliary signal lets the model adapt to previously unseen preference domains without retraining |

A Scene in a Coffee Shop

Imagine walking into a café you've never visited. You glance at the menu — latte, americano, oat milk, coconut milk — and hesitate for three seconds. The barista notices and asks: "Do you usually like your coffee milky or lighter?" You say "lighter." She recommends a cold drip. You love it.

Notice what happened. The barista did not ask you to fill out a ten-page preference questionnaire and spend three months training a taste-prediction model. She just watched your three seconds of hesitation, asked one question, and guessed almost perfectly.

That's how humans normally understand each other: we rarely rely on "static profiles." A glance, a pause, an offhand remark — these dynamic, contextual signals are what we actually use to read people.

Now consider AI alignment. The mainstream approach, RLHF (Reinforcement Learning from Human Feedback), does the opposite: it spends months collecting tens of thousands of human preference comparisons, trains a static reward model, and bakes that "taste predictor" permanently into the AI. Changed your mind? The model doesn't know. You belong to a culture never covered by annotators? It knows even less. This paper asks a simple but sharp question: what if AI could, like that barista, infer your preferences quickly from a handful of dynamic signals?

The Hidden Cracks in RLHF

How RLHF works:

1. Collect comparisons. Human annotators see two model responses and pick the better one, according to project-defined criteria (helpfulness, honesty, harmlessness). 2. Train a reward model that scores any text — higher means more aligned with human preferences. 3. Fine-tune the language model with reinforcement learning to produce text the reward model scores highly.

This pipeline shaped today's major AI systems. But it has cracks:

• Crack one: the sea of preferences is far larger than the training pool. Annotators come mostly from a specific demographic (largely US-based, college-educated, English-fluent). Their preferences are not humanity's. A farmer in India's idea of a "good answer" may differ sharply from a Silicon Valley engineer's — but the reward model has never "seen" the farmer.
• Crack two: preferences change; models don't. Values evolve over time, but a static reward model is frozen once trained. Without costly re-annotation and retraining, it stays in the past.
• Crack three: multi-reward frameworks still box themselves in. Frameworks with separate models for "helpful," "harmless," and "humorous" remain confined to predefined preference domains. If a genuinely new preference type appears tomorrow, they are helpless.

In-Context Reward Adaptation: Preferences as an Inference Problem

The proposed method, In-Context Reward Adaptation (ICRA), rests on a clean intuition: if a transformer can learn to translate an unseen language in context, why can't it learn to understand an unseen preference in context?

How it works:

1. Give it a few examples — not tens of thousands, but a handful (e.g., three to five pairs of "response A vs. response B" with the human's choice). 2. Let the transformer treat these as an inference task: "Based on these comparisons, infer the reward function behind these choices." 3. Score new text with the inferred reward function — no weight updates, no gradient descent, no lengthy training.

But it isn't that simple. The authors prove an alarming theoretical result: standard transformers, in the asymptotic sense, deviate from the true reward function — even with more and more examples, the inference does not converge to the correct answer. There is always a systematic bias.

It's like a barista who hears you describe three coffees and, every time, interprets you as liking things "sweeter" when you actually like them "tartter." The more you describe, the more confident she becomes in the wrong direction — the bias doesn't shrink; it gets reinforced.

Response Time: An Overlooked Gold Mine

How to fix the bias? The authors' answer is surprising: look at how long humans took to choose.

This draws on a classic psychological intuition: when you hesitate between two options, your reaction time leaks information. If you prefer lattes to americanos, you may choose instantly. If you agonize for five seconds between a latte and a cappuccino, the gap in your mind is small — or your preference is unstable.

ICRA feeds human response time into the transformer as an auxiliary input signal. According to the paper, this single modification enables the model to successfully adapt to preferences from previously unseen domains.

Why does it work? Response time adds a dimension of preference "strength" and "certainty." A binary label "chose A" can't tell the model whether A won by a landslide or barely. Instant choices signal strong preferences; hesitation signals weak or ambiguous ones. This extra signal breaks the standard transformer's asymptotic bias, letting inference converge toward the true reward structure. The elegance: it doesn't add model complexity — it introduces a richer data representation, treating human hesitation itself as information.

The Theoretical Anchor

The paper's real weight comes from its theory. The authors characterize the asymptotic bias of standard transformers, proving why binary choice labels alone are insufficient to recover the underlying reward function. This matters independently: it tells us in-context learning is not a panacea. Some tasks require extra signals to anchor the model.

Response time serves as that anchor — a first-order approximation of preference strength. That said, the authors do not give a full mathematical account (in the abstract) of precisely how the bias is eliminated, to what degree, or under what conditions it fails. I could not access the full PDF, so I cannot judge the completeness or rigor of the theoretical derivation. That is a gap I must honestly flag.

Why This Matters

Today's AI alignment is, at bottom, a contest over whose values get encoded and whose get ignored. RLHF data comes from a narrow population, so AI behavior tilts toward that population — not out of malice, but statistical inevitability: your training data is your bias.

ICRA points elsewhere: rather than pre-encoding all possible human preferences, give AI the ability to read preferences on the spot. If this path works:

• Personalized alignment. One model facing a conservative elder and an iconoclastic youth could adjust its behavioral boundaries automatically — not from two rule sets, but by "reading" each user's preference domain within the first few turns.
• Cross-cultural adaptation. An AI trained on North American data, deployed in Africa or South Asia, could recalibrate its value judgments from a small sample of local feedback — no retraining, no costly localization annotation.
• Dynamic evolution. As social values shift, AI could adapt incrementally rather than requiring a multi-million-dollar "re-alignment" every couple of years.

But shadows exist: the abstract contains no concrete experimental results. We don't know how many preference domains ICRA was tested on, how much it improves over static reward models, or how it performs in real conversations. Those numbers are essential for judging its practical value.

Limitations and Open Questions

1. The privacy cost of response time. Recording milliseconds per click means fine-grained behavioral tracking, raising serious privacy concerns at scale. The paper doesn't discuss balancing signal acquisition against privacy protection. 2. Defining "unseen domains." How unseen is "previously unseen"? A new subculture, or an entirely different civilizational framework? If domains differ enough (e.g., from "efficiency-seeking" to "ritual-seeking"), are a few response-time-labeled examples still enough? No analysis of domain-difference magnitude appears in the abstract. 3. Manipulation risk. If a malicious user knows the AI infers preferences from response times, could they fake hesitation — deliberately delaying clicks — to distort the inferred weights? Where are ICRA's safety boundaries? Unanswered. 4. Compatibility with existing RLHF pipelines. Does ICRA replace or augment the static reward model? Is it a test-time dynamic adaptation layer or a training-time data enrichment step? These engineering details determine deployability.

A Metaphor and Its Boundary

Imagine giving a talk at an international conference with listeners from a dozen countries. The traditional approach: survey everyone beforehand and prepare an "averaged" speech — offending no one, moving no one. That's a static reward model.

ICRA's approach: after you take the stage, tell two short jokes and watch the front row — who laughs, who frowns, who pulls out their phone. From seconds of observation, you adjust your tone and content on the fly. You didn't prepare twelve speeches; you read the audience live.

The metaphor has limits: real human communication relies on more than response time. A frown might mean disagreement — or a stomachache. ICRA currently uses only one dynamic signal; whether it can extend to richer multimodal signals (facial expressions, tone, interaction history) is an open direction.

The Bigger Picture

While the AI community has been fixated on scale — bigger models, more data, longer training — ICRA points to a different dimension: smarter signals, not more signals. It echoes cognitive science's old idea of sparse coding: the brain doesn't store every image, just a few key basis vectors for reconstructing everything. ICRA seems to do something similar in preference space: learn a meta-structure of preference — the ability to infer a full preference map from a few examples.

If pursued, we might see a fundamental shift in alignment: from "pre-programming values" to "negotiating values on the spot." AI would no longer be a missionary with a fixed moral checklist, but a collaborator that continuously learns and co-shapes behavioral boundaries with you.

That is both exciting and unsettling. Exciting because of the flexibility; unsettling because an AI that reads human preferences too well may also become too good at flattering — saying what you want to hear rather than what you need to hear. That is the coin ICRA leaves us, gleaming on both sides.

References

1. Sun, Z., Xu, Z., & Wei, E. (2026). *In-Context Reward Adaptation for Robust Preference Modeling*. arXiv:2605.30323. 2. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. *NeurIPS 2022*. (Original RLHF paper) 3. Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. *NeurIPS 2023*. (DPO) 4. Brown, T., et al. (2020). Language Models are Few-Shot Learners. *NeurIPS 2020*. (Foundation of in-context learning) 5. For psychological work on response time in preference judgments, see Kahneman's heuristics-and-biases research program.