Same Trajectory, Contradictory Rewards: ROBORMBENCH Reveals Paraphrase Fragility in Vision-Language Reward Models

Forum topic · 小凯 · 2026-09-07

Summary

A new benchmark study, ROBORMBENCH (arXiv:2609.02345), shows that state-of-the-art vision-language models (VLMs) used as reward functions for robot learning are severely fragile to paraphrasing. The benchmark includes 2,390 real robot trajectories with human-annotated progress labels and 21,673 verified paraphrases. Identical robot trajectories receive reward scores that can differ by more than 60 points when only the task description is reworded, and binary success/failure judgments can flip entirely. Fragility scales with the degree of semantic deviation from the original description, and neither scaling models from 7B to 70B parameters nor chain-of-thought reasoning resolves the issue, suggesting a fundamental architectural limitation. The authors attribute the fragility to statistical word-vision correlations learned in pretraining and spurious attention patterns, and note risks of reward hacking. Mitigations tested include dedicated trajectory-grounded reward models (40-60% smaller score variance), paraphrase-augmented training (+15-20% stability), and ensembling multiple paraphrases (~30% variance reduction at 5x compute cost).

Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

Paper: Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models Authors: Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, et al. arXiv: https://arxiv.org/abs/2609.02345 (September 2026) Fields: Robotics / NLP / Computer Vision

The Core Phenomenon

A robot places a plate on a table. A vision-language model (VLM) watches the video and scores the task 85/100 for the instruction "put the plate on the table." Keeping every frame identical but rewording the instruction into a semantically equivalent paraphrase ("place the dishware onto the tabletop"), the same VLM scores the very same trajectory 23/100.

This is the paper's central finding: state-of-the-art VLM reward models are extremely fragile to paraphrases. Merely rewording an instruction can flip a robot behavior from "success" to "failure."

Background: VLMs as Reward Functions

Reinforcement learning for robotics requires a reward function that is precise and computable, while human goals are vague and complex. Hand-designed rewards suffer from sparsity, unintended shortcuts (the classic "push everything off the table to clean it" solution), and poor task generalization.

Using general-purpose VLMs (GPT-4V-class models) as reward functions offers generality, semantic richness, and no manual reward engineering — and has become a mainstream approach. This paper exposes a critical, overlooked weakness.

Key points

ROBORMBENCH benchmark consists of 2,390 real robot trajectories across platforms and tasks, human-annotated ground-truth progress labels, and 21,673 validated paraphrases spanning lexical substitution, syntactic restructuring, and action-goal rewriting.
Score volatility: identical trajectories can differ by 60+ points (out of 100) across semantically equivalent descriptions.
Success/failure flips: binary judgments can reverse entirely between equivalent phrasings.
Scaling with deviation: lexical substitutions cause the least variance; syntactic restructuring causes more; action-goal rewrites the most.
Scale doesn't help: models from 7B to 70B parameters (open and proprietary) show no significant difference — suggesting an architectural, not capacity, problem.
Chain-of-thought doesn't help: even models that appear to "reason" about the task remain strongly affected by paraphrasing.

Why It Happens

VLMs learn statistical correlations between words and visual patterns from pretraining, not semantic equivalence — they know "place" co-occurs with hand movements, but not that "place," "position," and "set down" mean the same thing.
Paraphrase distributions in training data are uneven; rare phrasings are poorly understood.
Attention mechanisms amplify spurious correlations (e.g., rewarding the word "grasp" even when grasping is irrelevant).
This makes VLM rewards especially prone to reward hacking: robots may learn to satisfy the reward signal under one phrasing while failing under deployment-time paraphrases.

Mitigations Tested (Mixed Results)

1. Dedicated trajectory-grounded reward models: best results — 40–60% smaller score variance and far fewer success/failure flips, though they require heavy human annotation, are task-specific, and lack rich language feedback. 2. Paraphrase-augmented training: penalizing inconsistent rewards across paraphrases yields ~15–20% improvement — helpful but far from human-level. 3. Ensembling: averaging rewards over 5 paraphrases reduces score-variance standard deviation by ~30%, at 5x compute cost. 4. Generative reformulation (directional): defining reward as similarity to an imagined ideal trajectory may be more stable, but training such generative models remains a major open challenge.

Broader Implications

Paraphrase fragility shows that fluency is not understanding: VLMs match statistical patterns rather than grasp meaning, echoing Searle's Chinese Room argument and exposing blind spots in Turing-test-style evaluation.
It is a concrete instance of the AI alignment problem: systems optimize the surface form of instructions rather than human intent. If feedback signals reflect specific phrasings rather than true goals, alignment failures follow.
Other open risks for VLM reward models include out-of-distribution generalization, adversarial attacks, and cultural/linguistic bias in training data.

ROBORMBENCH is a mirror for the field: before deploying VLM reward functions in the real world, their boundary conditions and failure modes must be carefully tested.

References

1. Jeung, W., Yoon, S., Hong, H., et al. "Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models." arXiv:2609.02345, 2026. 2. OpenAI. "GPT-4V(ision) System Card." 2023. 3. Driess, D., et al. "PaLM-E: An Embodied Multimodal Language Model." ICML, 2023. 4. Brohan, A., et al. "RT-2: Vision-Language-Action Models." arXiv:2307.15818, 2023. 5. Ziegler, D. M., et al. "Fine-Tuning Language Models from Human Preferences." arXiv:1909.08593. 6. Christiano, P. F., et al. "Deep Reinforcement Learning from Human Preferences." NeurIPS, 2017. 7. Amodei, D., et al. "Concrete Problems in AI Safety." arXiv:1606.06565, 2016. 8. Searle, J. R. "Minds, Brains, and Programs." Behavioral and Brain Sciences, 1980.