From Imitation to Trial-and-Error: How Reinforcement Learning Helps VLA Models Break Past Rote Memorization

*An analysis of the paper “Can RL Improve Generalization of LLM Agents? An Empirical Study”*

Introduction: When AI Becomes an “Exam-Cramming Student”

Imagine a student who memorized the entire textbook and answers every practice problem perfectly — but freezes when the same concepts appear in a new context. This is the predicament facing today's large language model (LLM) agents.

Most VLA (Vision-Language-Action) models are trained with supervised fine-tuning (SFT): the model imitates recorded human demonstrations, learning pattern matching of the form “see A, do B” rather than genuine understanding. When conditions change even slightly, performance collapses.

Researchers from Fudan University, Meituan, and Shanghai AI Laboratory argue that reinforcement learning (RL) is the key to moving from rote memorization to true generalization.

Background: The Glass Ceiling of SFT-Trained VLA Models

VLA models can *see* (process images/video), *read* (understand natural-language instructions), and *act* (execute actions in real or simulated environments). Their SFT-based training has a fatal weakness: distribution shift. Minor visual changes (lighting, angle, background) or semantic changes (different phrasings of instructions) are enough to severely degrade performance, because the model never saw such variations in training data.

The Study: A Three-Dimension Generalization Benchmark

The team built a benchmark testing OpenVLA-7B across three dimensions:

• Visual generalization: varying lighting, camera angles, backgrounds, object textures/colors
• Semantic generalization: paraphrased or more complex instructions (“grab that red thing” vs. “hand me the apple”)
• Execution generalization: different initial/target positions, obstacle configurations, physical parameters

They compared PPO (Proximal Policy Optimization), DPO (Direct Preference Optimization), and GRPO (Group Relative Policy Optimization) applied after SFT.

Key Findings

• PPO clearly outperformed DPO and GRPO, especially on semantic understanding and execution robustness.
• Why PPO wins:
1. Better exploration via policy entropy rewards, essential for covering diverse visual/semantic conditions. 2. More stable training through importance-sampling-ratio clipping, avoiding catastrophic forgetting. 3. Denser reward signals — per-step feedback rather than only end-of-episode success/failure, which matters for fine-grained motor control.
• RL did not meaningfully improve visual generalization. Perceptual robustness appears to stem from model architecture and pretraining; addressing it likely requires data augmentation or domain randomization at the pretraining stage, not fine-tuning-stage RL.

Practical PPO Training Recipe

• Batch size: sample 8–16 responses per prompt for reliable gradient estimates
• Learning rate: 1e-6 to 5e-6 (avoid destroying pretrained knowledge)
• Reward scaling: normalize rewards to [-1, 1]; boost magnitude for low-success-rate tasks
• Entropy coefficient: 0.01–0.05, gradually decayed during training
• Reward design: combine success bonuses, progress rewards (distance reduction), collision/failure penalties, and small step penalties to encourage efficiency
• Curriculum learning: train on easy scenarios first, then progressively introduce harder visual, semantic, and execution variations

Implications

• RL is not optional. For generalization, SFT alone is insufficient; a “SFT for initialization, RL for generalization” paradigm is emerging.
• PPO remains a strong baseline. Newer methods like DPO and GRPO should not be adopted blindly; method choice should follow task characteristics.
• Better evaluation is needed. The proposed visual/semantic/execution framework fills a gap left by success-rate-only benchmarks.
• Open directions: improving visual generalization (domain randomization, better visual pretraining), more sample-efficient or offline RL, and cross-task transfer (meta-learning).

Conclusion

RL lets VLA agents move beyond imitating surface behavior toward adapting to new instructions and conditions. It is not a silver bullet — visual generalization remains unsolved — but it opens a new frontier: agents that can handle the complexity of the real world, not just standardized test questions.

References

1. Zhou, Y., Chen, Y., Li, X., et al. (2025). Can RL Improve Generalization of LLM Agents? An Empirical Study. *arXiv preprint arXiv:2503.XXXXX*. 2. Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. 3. Rafailov, R., et al. (2023). Direct Preference Optimization. NeurIPS 2023. 4. Kim, G., et al. (2025). OpenVLA: An Open Vision-Language-Action Model. 5. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.