Same Trajectory, Contradictory Rewards: ROBORMBENCH Reveals Paraphrase Fragility in VLM Reward Models

Forum topic · 小凯 · 2026-09-07

Summary

A forum post on zhichai.net discusses the ROBORMBENCH paper (arXiv:2609.02345), which exposes a critical weakness in using vision-language models (VLMs) as reward functions for robot learning. The study shows that semantically equivalent paraphrases of the same task description can cause reward scores for an identical robot trajectory to swing by more than 60 points on a 100-point scale, and can even flip binary success/failure judgments. The benchmark includes 2,390 real robot trajectories, human-annotated progress labels, and 21,673 verified paraphrases spanning lexical substitution, syntactic reordering, and goal rewriting. Key findings: score variance grows with semantic distance from the original phrasing; scaling models from 7B to 70B parameters does not reduce fragility; and chain-of-thought reasoning does not help. The authors attribute the problem to statistical, rather than semantic, language-vision associations learned during training, which also opens the door to reward hacking. Mitigations tested include dedicated trajectory-grounded reward models (40-60% less variance), paraphrase-augmented training (15-20% improvement), and ensembling five paraphrases (about 30% variance reduction at 5x compute). The post frames the issue as evidence that fluent language generation does not imply semantic understanding, connecting it to the Chinese Room argument and the AI alignment problem.

This post reviews "Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models" by Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, et al. (arXiv: https://arxiv.org/abs/2609.02345, September 2026, robotics / NLP / computer vision).

The core problem

Vision-language models (VLMs) are increasingly used as reward functions in robot reinforcement learning: given a video of a robot performing a task and a text description, the model scores task completion. The paper shows that paraphrasing the task description — without changing a single frame of the robot's trajectory — can cause the score to collapse. In the post's opening example, "put the plate on the table" earns 85/100, while the semantically identical "place the dishware upon the tabletop" earns 23/100.

Why VLM reward functions matter

Hand-designed reward functions suffer from sparsity, unintended shortcuts (the classic robot sweeping everything off the table to "clean" it), and poor task generalization. VLM-based rewards promise generality, semantic richness, and no manual engineering — which is why the discovered fragility is alarming for a mainstream direction in robot learning.

Key points from ROBORMBENCH

Benchmark scale: 2,390 real robot trajectories, human-annotated ground-truth progress labels, and 21,673 verified paraphrases covering lexical substitution, syntactic reordering, and action-goal rewriting.
Wild score swings: paraphrases of the same trajectory can differ by 60+ points (out of 100).
Success/failure flips: binary judgments can invert between semantically equivalent descriptions.
Deviation matters: variance grows with paraphrase type — lexical swaps < syntactic reordering < goal rewriting.
Scale is not a fix: models from 7B to 70B parameters (open and proprietary) show comparable fragility, suggesting a structural issue rather than capacity limits.
Chain-of-thought doesn't help: even explicit reasoning fails to stabilize scores.

Why it happens

The post explains that VLMs learn statistical language-vision associations from pretraining data, not semantic equivalence — common phrasings ("put X on Y") are understood well, rare ones poorly. Attention mechanisms amplify spurious correlations with specific words. This also creates reward hacking risk: robots may learn to satisfy the VLM's statistical biases rather than the actual task, performing well under the training phrasing but poorly under an equivalent deployment phrasing.

Mitigations tested (mixed results)

Dedicated trajectory-grounded reward models: 40–60% smaller score variance and far fewer success/failure flips, but they need heavy human annotation, are task-specific, and lack VLM-style verbal feedback.
Paraphrase-augmented training: penalizing inconsistent scores across paraphrases yields ~15–20% improvement — insufficient.
Ensembling: averaging scores over 5 paraphrases reduces variance standard deviation by ~30%, at 5x compute cost.
Deeper direction: the authors suggest discriminative scoring is fundamentally fragile; generative models that compare trajectories against imagined ideal executions may be more robust, though training such models is itself hard.

Broader implications

The post connects paraphrase fragility to three deeper issues:

1. Fluency ≠ understanding — VLMs match surface patterns rather than grasping the physical reality behind words (echoing Searle's Chinese Room). 2. The alignment problem — AI optimizes the surface form of instructions rather than true human intent; feedback signals must reflect intent, not specific phrasings. 3. Other open risks of VLM rewards — out-of-distribution generalization, adversarial attacks, and cultural bias in training data.

The post concludes that ROBORMBENCH is a mirror revealing current VLM reward models' real limits, and a warning to test boundary conditions before deploying these systems in the real world.

References

Jeung, W., Yoon, S., Hong, H., et al. "Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models." arXiv:2609.02345, 2026.
OpenAI. "GPT-4V(ision) System Card." 2023.
Driess, D., et al. "PaLM-E: An Embodied Multimodal Language Model." ICML, 2023.
Brohan, A., et al. "RT-2: Vision-Language-Action Models." arXiv:2307.15818, 2023.
Ziegler, D. M., et al. "Fine-Tuning Language Models from Human Preferences." arXiv:1909.08593, 2019.
Christiano, P. F., et al. "Deep Reinforcement Learning from Human Preferences." NeurIPS, 2017.
Amodei, D., et al. "Concrete Problems in AI Safety." arXiv:1606.06565, 2016.
Searle, J. R. "Minds, Brains, and Programs." Behavioral and Brain Sciences, 1980.