论文信息
- Title: Reinforcement Learning without Ground-Truth Solutions can Improve LLMs
- Authors: Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang
- Area: Machine learning (cs.LG)
- Scale dominance: uncalibrated score magnitudes distort policy updates. A problem scored in the 1000–2000 range produces gradient signals a million times stronger than one scored 0.001–0.002, so the model gravitates toward problems with "louder" reward landscapes rather than more important ones.
- Frequency dominance: with 95 mediocre candidates and 5 excellent ones, gradient updates are dominated by the mediocre majority, drowning out the rare but stronger solutions.
- Instance-wise ranking gives scale invariance: a rank of 1st means the same thing whether the underlying score is 1500 or 0.0015, so no problem dominates learning due to score magnitude.
- Preserving feedback for valid-but-lower-ranked solutions prevents frequency dominance: strong solutions receive disproportionately high reward rather than being averaged away.
- ALE-Bench: Qwen3-8B +8.9%, GLM-Z1-9B-0414 +9.4%
- LiveCodeBench: +2.4% average
- USACO: +3.5% average
- RLVR's reach expands from "has a ground-truth answer" to merely "has a score." Most real-world optimization, design, and creative problems can be scored but not verified; all of these now become trainable.
- Strategy over memorization: because training provides no answers to memorize, the model is forced to learn how to analyze, try, and compare — a transferable skill.
- Lin, Y., Gao, Q., & Kuang, N.L. (2026). *Reinforcement Learning without Ground-Truth Solutions can Improve LLMs*. arXiv preprint.
- Shao, Z., et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv preprint.
- Guo, S., et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv preprint.
- Ouyang, L., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022.
- Silver, D., et al. (2017). *Mastering the game of Go without human knowledge*. Nature, 550, 354–359.
---
The problem: RLVR needs standard answers
Reinforcement Learning with Verifiable Rewards (RLVR) — the core technique behind models like DeepSeek-R1 and OpenAI o1 — depends on a deterministic verifier: unit tests for code, proof checkers for math. But many real-world problems (algorithm optimization, engineering design, creative work) have no ground-truth answer, or verifying correctness is harder than solving the problem. A natural question follows: if no one can grade the AI, can it still learn?
Raw continuous scores introduce two further failure modes:
RiVER: rank instead of score
RiVER's key idea: compare candidates relative to each other, not against an absolute standard. Its pipeline:
1. Sample a group of candidate answers per problem instance (e.g., 16). 2. Rank them within the instance by their scores. 3. Shape rewards from rank: top-ranked candidates get large positive rewards; lower-ranked ones get small or negative rewards — but valid solutions are never zeroed out entirely. 4. Update the policy with GRPO/PPO-style methods on these calibrated rewards.
Two design choices matter:
Formally, the shaped reward is \(R_i = f(rank_i, N)\), a monotone function of the candidate's in-group rank. Compared to GRPO's binary correct/incorrect rewards, this continuous, ordinal signal works even when no "correct answer" exists.
The intuition mirrors human "ordinal learning": we learn from relative comparisons ("this attempt was better than the last") rather than absolute scores — a baby learns to walk by comparing attempts, not from a gait score of 73.5.
Experiments: training without answers, improving on tasks with answers
The authors trained exclusively on 12 AtCoder Heuristic Contest (AHC) problems — optimization tasks scored by a metric with no known optimum — then tested on exact-answer benchmarks:
The transfer to LiveCodeBench and USACO is striking: the model never saw binary correct/incorrect feedback during training, yet its learned reasoning and problem-decomposition abilities carried over.
A crucial control: a baseline trained on raw execution scores improved on ALE-Bench (the training distribution) but failed to transfer to LiveCodeBench and USACO. This shows the ranking-based calibration — not mere exposure to scored tasks — is what produces transferable capability. The model learns a meta-strategy for solving problems rather than memorizing answers.
Why it matters
Potential applications include open-ended scientific discovery (e.g., proposing novel material structures evaluated by simulation), creative industries optimized by outcome metrics like click-through rates, and personalized assistants that improve from relative user feedback.
Conclusion
RiVER demonstrates that knowing what is better can substitute for knowing what is right. By converting absolute scores into instance-wise rankings, reinforcement learning works in arenas with no referee, no scoreboard, and no finish line — and the abilities learned there even improve performance on tasks that do have correct answers.
---