RiVER: Reinforcement Learning Without Ground-Truth Answers Can Improve LLMs

Forum topic · 小凯 · 2026-06-26

Summary

This post introduces RiVER (Ranking-induced VERifiable framework), a reinforcement learning method that trains large language models on tasks lacking ground-truth solutions. Traditional RLVR (RL with Verifiable Rewards), used to train models like DeepSeek-R1 and OpenAI o1, requires deterministic verifiers such as unit tests or proof checkers—unavailable for open-ended optimization, design, and engineering problems. RiVER replaces raw scores with instance-wise relative rankings within candidate groups, solving two failure modes of naive score-based RL: 'scale dominance' (un calibrated score magnitudes across problems distort policy updates) and 'frequency dominance' (many mediocre samples outweigh rare strong ones). The framework builds on GRPO, assigning calibrated rewards based on rank position while still rewarding valid but suboptimal candidates. Experiments trained Qwen3-8B and GLM-Z1-9B-0414 on 12 AtCoder Heuristic Contest problems (optimization tasks with no correct answer). Gains transferred to verifiable benchmarks: ~8.9-9.4% improvement on ALE-Bench, 2.4% on LiveCodeBench, and 3.5% on USACO. A raw-score baseline trained on the same data failed to transfer to exact-solution benchmarks, showing ranking-based calibration is the key driver of generalizable reasoning. RiVER effectively lowers RLVR's requirement from 'having a correct answer' to 'having a score,' opening RL training to engineering, scientific discovery, and creative domains.

RiVER: RL Without Ground-Truth Answers Can Improve LLMs

This post discusses the paper "Reinforcement Learning without Ground-Truth Solutions can Improve LLMs" by Yingyu Lin, Qiyue Gao, and Nikki Lijing Kuang (cs.LG, June 2026).

Key points

The problem with RLVR: RL with Verifiable Rewards — the paradigm behind DeepSeek-R1 and OpenAI o1 — depends on deterministic verifiers (unit tests, proof checkers). Most real-world problems (algorithm optimization, engineering design, creative work) have no ground-truth answer, and verification can be harder than solving.
Two failure modes of naive score-based RL:
Scale dominance: raw score magnitudes differ wildly across problems (e.g., milliseconds vs. seconds), so gradient updates from one problem dwarf another's by orders of magnitude.
Frequency dominance: many mediocre candidates dominate gradient updates, drowning out rare but strong solutions.
RiVER's solution: use instance-wise relative rankings instead of raw scores. For each problem, sample a group of candidates (e.g., 16), rank them by score, and assign calibrated rewards by rank position ( $$R_i = f(rank_i, N)$$ , with $$f$$ monotone decreasing). Top-ranked candidates get strong positive rewards; valid but lower-ranked solutions still get partial credit. Building on GRPO, RiVER shifts from binary (correct/incorrect) to continuous ranking-based rewards, achieving scale invariance across problems.
Why rankings work: ranking is scale-invariant — first place is first place regardless of the score scale — echoing humans' natural "ordinal learning," where we compare outcomes relatively rather than against absolute scores.

Experimental setup

Training: 12 AtCoder Heuristic Contest (AHC) problems — optimization tasks with scores but no known optimum, i.e., no ground truth.
Evaluation: benchmarks that *do* have ground-truth answers — ALE-Bench, LiveCodeBench, USACO.

Findings

Qwen3-8B: +8.9% on ALE-Bench; GLM-Z1-9B-0414: +9.4%.
Cross-transfer to exact-solution tasks: +2.4% on LiveCodeBench, +3.5% on USACO — despite never seeing binary correct/incorrect feedback during training.
Critical baseline: training directly on raw execution scores improves ALE-Bench but fails entirely to transfer to LiveCodeBench/USACO, proving that ranking-based calibration is what produces generalizable reasoning rather than task-specific gains.

Why it matters

RiVER lowers RLVR's entry bar from "requires a verifiable correct answer" to "requires any score," opening RL training to engineering optimization, scientific discovery, creative generation, and personalized assistants.
Models trained without ground truth learn transferable meta-strategies (problem analysis, comparing alternatives) instead of memorizable answers — suggesting genuine learning comes from comparison and selection, not memorization.

References

Lin, Y., Gao, Q., & Kuang, N.L. (2026). *Reinforcement Learning without Ground-Truth Solutions can Improve LLMs*. arXiv preprint.
Shao, Z., et al. (2024). *DeepSeekMath*. arXiv preprint.
Guo, S., et al. (2025). *DeepSeek-R1*. arXiv preprint.
Ouyang, L., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022.
Silver, D., et al. (2017). *Mastering the game of Go without human knowledge*. Nature, 550, 354-359.