Imagine this: you're using ChatGPT to write a cover letter. The model produces a response, you frown, your mouse hesitates over the "regenerate" button for a few seconds, your gaze lingers on the third paragraph—and then you give no feedback at all and type your next instruction.
In traditional LLM alignment paradigms, that moment never happened. No like, no dislike, no rating. But the Haw-Shiuan Chang team at UMass Amherst says: wait—that hesitation and pause *was* feedback.
The Wasted 97%
A striking number: only 3% of users actively provide feedback (likes/dislikes), according to research by Wang et al. In other words, 97% of user experience information is being wasted.
Internet giants have long understood the value of implicit feedback—how long you linger on a product page, where your mouse hesitates, where your scrolling slows. Recommender systems live on this data. Yet LLM alignment relies almost entirely on explicit feedback (RLHF), forcing users to do something most people won't do: rate responses.
The team's core question: can we predict user preferences for LLM responses from mouse trajectories and eye-tracking data?
The IFLLM Dataset: 1,336 Conversations from 59 People
The researchers built a data collection website and recruited 59 Mechanical Turk workers, collecting 1,336 multi-turn Q&A conversations. For each user:
- Mouse trajectories were fully recorded (paths, dwell points, click patterns)
- Eye-tracking data was captured via webcam (fixations, fixation duration, saccade patterns)
- Users provided explicit preference labels after each turn (current response vs. previous one) as ground truth for training
- Text-only reward model (predicting preference from response content): 55% accuracy
- Reward model with implicit feedback: 64% accuracy
The data shows users preferred the current response in 70% of labels—users are generally satisfied with LLM responses, but the degree of satisfaction is hidden in their mice and eyes.
A key finding: users' gaze behavior and mouse trajectories vary enormously. Some stare, some skim rapidly, some move the mouse with their eyes, some wander. This diversity means no single "standard user behavior pattern" works—personalized modeling is needed.
From 55% to 64%: What 9 Points Is Worth
The team trained a reward model based on implicit feedback and compared it to a traditional text-only reward model:
The real killer app is downstream: when this implicit-feedback reward model is used for DPO (Direct Preference Optimization) training, relative response quality improvements nearly tripled across 8 different LLMs.
Threefold. Not 30%—3x.
What This Means
A restaurant analogy. The traditional approach: send customers a survey asking "how was the food?" Only 3% respond, and you improve the menu based on those 3%. That's RLHF today.
The new approach: put sensors at tables recording how much of each dish was eaten, where chopsticks hesitated, how many times they were set down. The 97% who don't fill out surveys don't matter—their chopsticks already voted.
This isn't futuristic. Taobao, TikTok, and YouTube have done this for years. The LLM alignment field is—surprisingly—a decade or more late.
Why Didn't Anyone Do This Before?
Not for lack of ideas—implicit feedback is hard:
1. Noisy: mouse wandering might mean the user is drinking coffee, not that the response is bad 2. High individual variation: some people move the mouse while reading, others don't at all 3. Special equipment: eye tracking previously required expensive hardware (Tobii, etc.) 4. Privacy concerns: camera data is sensitive
The paper's breakthrough: eye tracking works with an ordinary laptop webcam, mouse data is natively available in browsers, and noise/individual differences are modeled with ModernBERT and random forests. The technical bar finally dropped to "doable."
Honest Limitations
Fifty-nine people isn't a small sample for ML, but it's only moderate for behavioral science. Do MTurk workers' behavior patterns represent real users? Does the lab setting (knowing one is recorded) alter behavior? These remain open questions.
Also, the 70% preference rate biased toward "like the current response"—if uncorrected, the model may learn to predict "like" whenever the user isn't visibly annoyed. The paper mentions this, but the discussion feels insufficient.
The Bigger Picture
The paper points to a larger trend: LLM alignment is moving from "exam mode" to "observation mode".
Exam mode: the user is an examiner who actively prompts, rates, and corrects; the model is a passive student.
Observation mode: the user interacts naturally; the model (or its reward model) quietly observes and infers satisfaction from behavior.
The latter is how humans actually relate. You don't ask a friend to fill out a feedback form each time they help—you read their expression, tone, and hesitation.
When LLMs are deployed to billions of users, explicit feedback will always be scarce. But implicit feedback—every mouse move, every scroll, every pause—is unlimited. This paper proves those signals can be used effectively.
What's next? Finer-grained behavioral signals, I'd guess: typing rhythm (speed changes signaling confusion), page-switching patterns, even device sensor data. The "implicit feedback revolution" in LLM alignment is just beginning.
---
Paper: Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users Authors: Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari, Aryan Sajith, Hamed Zamani (UMass Amherst / York University) arXiv: https://arxiv.org/abs/2606.20482 Code and data: https://github.com/themehulpatwari/llm-implicit-feedback/