RiVER: Reinforcement Learning Without Ground-Truth Answers Can Improve LLMs

Forum topic · 小凯 · 2026-06-26

Summary

RiVER (Ranking-induced VERifiable framework) extends reinforcement learning with verifiable rewards (RLVR) to problems that have no ground-truth answers. Traditional RLVR, used to train models like DeepSeek-R1 and OpenAI o1, requires deterministic verifiers such as unit tests or proof checkers, and fails on open-ended optimization tasks. RiVER replaces absolute scores with instance-wise relative rankings of candidate solutions, solving two key failure modes: scale dominance (raw score magnitudes skewing gradient updates across problems) and frequency dominance (many mediocre candidates drowning out rare strong ones). The method builds on GRPO, using a rank-based calibrated reward function. Models (Qwen3-8B, GLM-Z1-9B-0414) trained only on AtCoder Heuristic Contest optimization problems—where no optimal answer exists—improved by 8.9% and 9.4% on ALE-Bench, and also transferred to exact-answer benchmarks, gaining 2.4% on LiveCodeBench and 3.5% on USACO, while raw-score baselines showed no transfer. The work lowers RLVR's requirement from 'having a correct answer' to 'having a score,' opening reinforcement learning to engineering design, creative generation, and scientific discovery domains.

Overview

This post introduces the paper "Reinforcement Learning without Ground-Truth Solutions can Improve LLMs" by Yingyu Lin, Qiyue Gao, and Nikki Lijing Kuang (machine learning, June 2026), which presents RiVER (Ranking-induced VERifiable framework) — a method that lets large language models learn via reinforcement learning even when no ground-truth answer exists.

Background: the limits of RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) — the paradigm behind models like DeepSeek-R1 and OpenAI o1 — depends on deterministic verifiers: unit tests for code, proof checkers for math. This creates an existential dependency on ground truth. Many real problems (algorithm optimization, engineering design, creative writing, scientific hypotheses) have no correct answer, and verifying a solution can be harder than finding one.

Two further problems arise when using raw scores as rewards:

Scale dominance: score magnitudes vary wildly across problems (e.g., 1000–2000 ms vs. 0.001–0.002 s), so gradients from high-scale problems overwhelm others by factors up to ~10^6.
Frequency dominance: when 95 mediocre candidates outnumber 5 excellent ones, the mediocre group dominates gradient updates and drowns out the rare strong solutions.

The RiVER approach

RiVER's core idea: replace absolute scores with relative rankings.

1. For each problem instance, sample a group of candidate solutions (e.g., 16). 2. Rank candidates by their scores within the instance. 3. Assign calibrated rewards based on rank position: top-ranked candidates get large positive rewards; lower-ranked but still valid candidates retain small positive feedback rather than being discarded. 4. Update the policy with GRPO/PPO-style methods.

Key design properties:

Instance-wise ranking makes rewards scale-invariant — the same relative rank yields the same reward regardless of raw score magnitude, fixing scale dominance.
Calibrated reward shaping preserves feedback for valid but suboptimal solutions while amplifying top-ranked ones, fixing frequency dominance.
Formally, the reward is a monotone decreasing function of rank: $$R_i = f(rank_i, N)$$ .

The authors connect this to human "ordinal learning": humans learn primarily through relative comparison ("better than last time") rather than cardinal scores.

Experiments and results

Training: 12 AtCoder Heuristic Contest (AHC) optimization problems — a true no-ground-truth setting where solutions are scored but the optimum is unknown.

Evaluation (trained only on no-answer tasks, tested on answer-verifiable tasks):

ALE-Bench: +8.9% (Qwen3-8B), +9.4% (GLM-Z1-9B-0414) over baselines
LiveCodeBench: +2.4% average
USACO: +3.5% average

A raw-score baseline (using uncalibrated execution scores) improved on ALE-Bench but failed to transfer to LiveCodeBench and USACO — showing that RiVER's rank-based calibration, not mere exposure to scored environments, drives transferable general reasoning ability.

Why it matters

Lowers RLVR's entry bar from "having a verifiable answer" to "having a score," opening RL training to engineering design, heuristics, creative generation, and scientific research.
Encourages strategy learning over answer memorization: without ground truth to memorize, models learn meta-level problem-solving that transfers across task types.
Suggests future applications in open-ended scientific discovery, creative industries, and personalized lifelong-learning assistants.

Conclusion

RiVER shows that knowing *what is better* can matter more than knowing *what is correct*. By training in worlds without standard answers, models acquire transferable reasoning skills that improve even on tasks with exact answers — evidence that genuine understanding emerges from comparison and selection rather than memorization.

References

Lin, Y., Gao, Q., & Kuang, N.L. (2026). *Reinforcement Learning without Ground-Truth Solutions can Improve LLMs*. arXiv preprint.
Shao, Z., et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv preprint.
Guo, S., et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv preprint.
Ouyang, L., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022.
Silver, D., et al. (2017). *Mastering the game of Go without human knowledge*. Nature, 550, 354-359.