Imagine this scenario: you are using ChatGPT to write a cover letter. The model gives an answer, you frown, your mouse hesitates over the 'Regenerate' button for a few seconds, your gaze lingers unusually long on the third paragraph—and then, without giving any feedback at all, you type your next instruction.
In the traditional LLM alignment paradigm, nothing happened in that moment. No like, no dislike, no rating—your experience vanished into the void. But Haw-Shiuan Chang's team at UMass Amherst says: wait a moment—your hesitation and lingering *were* feedback.
The Wasted 97%
First, a number that makes you sit up: only 3% of users actively provide feedback (likes/dislikes), according to research by Wang et al. In other words, 97% of user experience information is being wasted.
Internet giants have long known the value of implicit feedback—how long you linger on Taobao, which product your mouse hesitates over, where your scroll speed slows down—these are gold mines. Recommender systems live on them. Yet LLM alignment relies almost entirely on explicit feedback (RLHF), forcing users to do something most people don't want to do: actively rate.
Chang's team posed a direct question: can we predict a user's preference for LLM answers from their mouse trajectories and eye movements?
The IFLLM Dataset: 1,336 Conversations from 59 People
To answer this, they built a data collection website and recruited 59 Mechanical Turk workers, collecting 1,336 multi-turn Q&A conversations. For each user while answering LLM questions:
- Mouse trajectories were fully recorded (movement paths, dwell positions, click patterns)
- Eye-tracking data was captured via webcam (fixation points, fixation duration, saccade patterns)
- Users gave an explicit preference annotation at the end of each round (current answer vs. previous answer), serving as training ground truth
- Text-only reward model (predicting preference from answer content): 55% accuracy
- Reward model with implicit feedback: 64% accuracy
The data shows that in 70% of annotations, users preferred the current answer—meaning users are generally satisfied with LLM answers, but the degree of satisfaction is hidden in their mice and eyes.
A key finding: users' gaze behavior and mouse trajectories vary enormously. Some people stare intently, some skim rapidly, some move their mice with their eyes, some have erratic mouse movement. This diversity means no single "standard user behavior pattern" can be applied—personalized modeling is required.
From 55% to 64%: What 9 Percentage Points Are Worth
The team trained a reward model based on implicit feedback and compared it with a traditional text-only reward model:
The real killer is downstream: when this implicit-feedback reward model was used for DPO (Direct Preference Optimization) training, relative answer-quality improvements nearly tripled across 8 different LLMs.
Threefold. Not 30%—3x.
What Does This Mean?
A restaurant analogy: the traditional approach is to send out surveys asking "was the food good?"—only 3% of customers fill them out, and you improve the menu based on those 3%. That's today's RLHF. The new approach installs sensors at the table, recording how much of each dish customers ate, where their chopsticks hesitated, how many times they set utensils down. The 97% who don't fill out surveys don't matter—their chopsticks already voted.
This isn't futuristic technology. Taobao, Douyin, and YouTube have been doing this for years. The LLM alignment field is—surprisingly—more than a decade late.
Why Didn't Anyone Do This Before?
It's not that no one thought of it—it was too hard. Implicit feedback has several thorny properties:
1. Noisy: erratic mouse movement may mean the user is drinking coffee, not that the answer is bad 2. Large individual differences: some people move their mouse while reading, others never touch it 3. Special equipment required: eye tracking used to need dedicated hardware (e.g., Tobii) costing thousands of dollars 4. Privacy concerns: webcam data is sensitive
The paper's breakthrough: eye tracking with an ordinary laptop webcam (accuracy just good enough), mouse data already native to the browser, and noise/individual differences modeled with ModernBERT and random forests. The technical barrier finally dropped to a "doable" level.
Honest Limitations
The paper has weaknesses. A sample of 59 people is not small for machine learning, but only medium-sized for behavioral science. Can MTurk workers' behavior represent real users? Does the lab setting (knowing they're being recorded) change behavior? These are open questions.
Also, the 70% preference rate skews toward "liking the current answer"—if unaddressed, the model might learn to predict "like" whenever the user doesn't visibly object. The paper mentions this, but in my view doesn't discuss it enough.
The Bigger Picture
This paper points to a larger trend: LLM alignment is shifting from "exam mode" to "observation mode".
Exam mode: the user is an examiner who actively asks, rates, and corrects. The model is a student passively receiving feedback. Observation mode: the user is a natural user. The model (or its reward model) silently observes and infers satisfaction from behavior.
The latter is how humans actually interact. You don't need friends to fill out a feedback form every time they help you—you know from their expressions, tone, and hesitation how to help better next time.
When LLMs are deployed to products with billions of users, explicit feedback will always be scarce. But implicit feedback—every mouse movement, every scroll, every pause—is infinite. This paper proves these signals can be effectively exploited.
What's next? My guess is finer-grained behavioral signals: keyboard typing rhythm (speed changes hinting at confusion), page-switching patterns, even device sensor data. The "implicit feedback revolution" in LLM alignment is just beginning.
---
Paper: Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users Authors: Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari, Aryan Sajith, Hamed Zamani (UMass Amherst / York University) arXiv: https://arxiv.org/abs/2606.20482 Code and data: https://github.com/themehulpatwari/llm-implicit-feedback/