English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

RiVER: Reinforcement Learning Without Ground-Truth Answers Can Improve LLMs

Forum topic · 小凯 · 2026-06-26

Summary

A new reinforcement learning framework called RiVER (Ranking-induced VERifiable framework) enables large language models to learn from tasks that have no ground-truth answers, according to a paper by Yingyu Lin, Qiyue Gao, and Nikki Lijing Kuang. Traditional RLVR (reinforcement learning with verifiable rewards) requires a deterministic verifier such as unit tests or proof checkers, which does not exist for open-ended optimization problems. RiVER replaces raw scores with instance-wise rankings of sampled candidate solutions, addressing two failure modes of raw-score training: scale dominance, where larger-magnitude reward signals skew policy updates across problems, and frequency dominance, where many mediocre samples drown out rare strong ones. Training only on 12 AtCoder Heuristic Contest problems (scored optimization tasks with no known optimum), RiVER-tuned Qwen3-8B and GLM-Z1-9B models improved ALE-Bench scores by 8.9% and 9.4%, and notably transferred to exact-answer benchmarks LiveCodeBench (+2.4%) and USACO (+3.5%), while a raw-score baseline failed to transfer. The results suggest ranking-based, scale-invariant reward shaping teaches transferable reasoning strategies rather than memorized answers, extending RL post-training to any domain where solutions can be scored but not verified.

论文信息

  • Title: Reinforcement Learning without Ground-Truth Solutions can Improve LLMs
  • Authors: Yingyu Lin, Qiyue Gao, Nikki Lijing Kuang
  • Area: Machine learning (cs.LG)
  • ---

    The problem: RLVR needs standard answers

    Reinforcement Learning with Verifiable Rewards (RLVR) — the core technique behind models like DeepSeek-R1 and OpenAI o1 — depends on a deterministic verifier: unit tests for code, proof checkers for math. But many real-world problems (algorithm optimization, engineering design, creative work) have no ground-truth answer, or verifying correctness is harder than solving the problem. A natural question follows: if no one can grade the AI, can it still learn?

    Raw continuous scores introduce two further failure modes:

  • Scale dominance: uncalibrated score magnitudes distort policy updates. A problem scored in the 1000–2000 range produces gradient signals a million times stronger than one scored 0.001–0.002, so the model gravitates toward problems with "louder" reward landscapes rather than more important ones.
  • Frequency dominance: with 95 mediocre candidates and 5 excellent ones, gradient updates are dominated by the mediocre majority, drowning out the rare but stronger solutions.
  • RiVER: rank instead of score

    RiVER's key idea: compare candidates relative to each other, not against an absolute standard. Its pipeline:

    1. Sample a group of candidate answers per problem instance (e.g., 16). 2. Rank them within the instance by their scores. 3. Shape rewards from rank: top-ranked candidates get large positive rewards; lower-ranked ones get small or negative rewards — but valid solutions are never zeroed out entirely. 4. Update the policy with GRPO/PPO-style methods on these calibrated rewards.

    Two design choices matter:

  • Instance-wise ranking gives scale invariance: a rank of 1st means the same thing whether the underlying score is 1500 or 0.0015, so no problem dominates learning due to score magnitude.
  • Preserving feedback for valid-but-lower-ranked solutions prevents frequency dominance: strong solutions receive disproportionately high reward rather than being averaged away.
  • Formally, the shaped reward is \(R_i = f(rank_i, N)\), a monotone function of the candidate's in-group rank. Compared to GRPO's binary correct/incorrect rewards, this continuous, ordinal signal works even when no "correct answer" exists.

    The intuition mirrors human "ordinal learning": we learn from relative comparisons ("this attempt was better than the last") rather than absolute scores — a baby learns to walk by comparing attempts, not from a gait score of 73.5.

    Experiments: training without answers, improving on tasks with answers

    The authors trained exclusively on 12 AtCoder Heuristic Contest (AHC) problems — optimization tasks scored by a metric with no known optimum — then tested on exact-answer benchmarks:

  • ALE-Bench: Qwen3-8B +8.9%, GLM-Z1-9B-0414 +9.4%
  • LiveCodeBench: +2.4% average
  • USACO: +3.5% average
  • The transfer to LiveCodeBench and USACO is striking: the model never saw binary correct/incorrect feedback during training, yet its learned reasoning and problem-decomposition abilities carried over.

    A crucial control: a baseline trained on raw execution scores improved on ALE-Bench (the training distribution) but failed to transfer to LiveCodeBench and USACO. This shows the ranking-based calibration — not mere exposure to scored tasks — is what produces transferable capability. The model learns a meta-strategy for solving problems rather than memorizing answers.

    Why it matters

  • RLVR's reach expands from "has a ground-truth answer" to merely "has a score." Most real-world optimization, design, and creative problems can be scored but not verified; all of these now become trainable.
  • Strategy over memorization: because training provides no answers to memorize, the model is forced to learn how to analyze, try, and compare — a transferable skill.
  • Potential applications include open-ended scientific discovery (e.g., proposing novel material structures evaluated by simulation), creative industries optimized by outcome metrics like click-through rates, and personalized assistants that improve from relative user feedback.

    Conclusion

    RiVER demonstrates that knowing what is better can substitute for knowing what is right. By converting absolute scores into instance-wise rankings, reinforcement learning works in arenas with no referee, no scoreboard, and no finish line — and the abilities learned there even improve performance on tasks that do have correct answers.

    ---

    References

  • Lin, Y., Gao, Q., & Kuang, N.L. (2026). *Reinforcement Learning without Ground-Truth Solutions can Improve LLMs*. arXiv preprint.
  • Shao, Z., et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv preprint.
  • Guo, S., et al. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv preprint.
  • Ouyang, L., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022.
  • Silver, D., et al. (2017). *Mastering the game of Go without human knowledge*. Nature, 550, 354–359.

Tags

#reinforcement-learning#llm#rlvr#river#reward-shaping#code-generation#atcoder#reasoning

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208160