When Can LLMs Learn to Reason with Weak Supervision? RLVR Saturation, Faithfulness, and the Eight-Example Miracle

Forum topic · 小凯 · 2026-04-21

Summary

A 2026 empirical study from UCLA, NYU, and Google (arXiv:2604.18574) examines whether reinforcement learning with verifiable rewards (RLVR) enables genuine generalization under three weak-supervision conditions: scarce data (as few as 8 examples), noisy rewards (up to 70% incorrect labels), and self-supervised proxy rewards (majority voting, self-certainty). Using GRPO across Qwen and Llama model families, the authors find that generalization depends on the pre-saturation phase of training: Qwen2.5-Math-1.5B saturated after 302 steps and gained 29.7% on MATH-500 from just 8 training examples, while Llama3.2-3B-Instruct saturated in 55 steps with far weaker gains, despite showing higher raw output diversity. Introducing reasoning faithfulness and faithful diversity metrics, the study shows Llama's diversity is largely unfaithful noise. A 2x2 intervention (continual pre-training x SFT) reveals that thinking-style SFT with explicit chain-of-thought is necessary for weak-supervision RL to work, while domain continual pre-training acts as an amplifier. Proxy rewards mostly cause reward hacking and collapse. Practical takeaways: treat the saturation step as a diagnostic signal, and invest compute in chain-of-thought SFT before scaling RL.

When Can LLMs Learn to Reason with Weak Supervision?

> Paper: When Can LLMs Learn to Reason with Weak Supervision? > arXiv: https://arxiv.org/abs/2604.18574 > Authors: Salman Rahman, Jingyan Shen, Anna Mordvina, Hamid Palangi, Saadia Gabriel, Pavel Izmailov > Institutions: UCLA, NYU, Google > Published: April 21, 2026

This post is a full English translation of a Chinese forum deep-dive on the paper.

Background: RLVR and Three Kinds of Weak Supervision

RLVR (Reinforcement Learning with Verifiable Rewards) trains LLMs with only a binary right/wrong signal on final answers — no step-by-step labels. It powers reasoning models like DeepSeek-R1, Kimi k1.5, and OpenAI's o1 series. The paper uses GRPO (Group Relative Policy Optimization), which compares a group of sampled answers per question instead of using a critic model.

The study examines three realistic weak-supervision settings:

1. Scarce data — only 8 training examples, reused across batches. 2. Noisy rewards — 30% or even 70% of labels are wrong. 3. Self-supervised proxy rewards — no ground truth at all; the model scores itself via majority voting or self-certainty.

Finding 1: The Pre-Saturation Phase Predicts Generalization

Training reward follows two phases: a pre-saturation phase where training reward climbs and test performance improves in tandem, and a post-saturation phase where reward plateaus and nothing further is learned. The key insight: the saturation step, not final numbers, reveals whether a model is learning or memorizing.

Qwen2.5-Math-1.5B, trained on just 8 MATH examples, saturated after 302 steps, gaining 29.7% on MATH-500 and 10.5% on out-of-domain SCP-Hard. Qwen2.5-Math-7B trained on graphs gained 21.0% on MATH-500 (OOD).
Llama3.2-3B-Instruct saturated after only 55 steps, gaining just 10.8% — it quickly memorized the 8 answers rather than learning transferable reasoning.

The counterintuitive pattern: faster saturation predicted worse generalization. Moreover, a "Large-Small Gap" metric comparing 2048-sample vs 8-sample training shows the gap is negligible in most model-domain pairs — during pre-saturation, learnable generalization barely depends on data quantity.

Finding 2: High Diversity Can Mask Cheating

Using LLM-as-a-judge (Gemini 3 Flash) with a Shannon Diversity Index, the authors found Llama maintains *higher* raw output diversity than Qwen throughout training — yet generalizes worse. The resolution is reasoning faithfulness: whether the chain of thought logically supports the final answer.

On MATH, Llama's faithfulness is significantly lower than Qwen's: many of Llama's "correct" answers are lucky guesses with incoherent derivations.
A refined metric, faithful diversity (diversity counted only over logically sound answers), erases Llama's apparent diversity advantage. Qwen explores genuinely different valid reasoning paths; Llama's variety is mostly noise.

Finding 3: Thinking SFT Installs Faithfulness — a 2×2 Intervention

Starting from Llama3.2-3B-Base, the authors crossed two factors:

CPT: none vs. ~52B math tokens (Nemotron-CC-Math)
SFT: Non-Thinking (answers only, ~0.27B tokens) vs. Thinking (full chains of thought from OpenThoughts-114K, ~1B tokens) — using the identical 43.5K problems

Results across all three weak-supervision settings:

Thinking SFT is necessary. Only Thinking-SFT models show meaningful generalization; Non-Thinking SFT models stay flat and are vulnerable to reward hacking.
CPT amplifies but cannot substitute. CPT + Thinking SFT is best overall; CPT + Non-Thinking SFT still fails — 52B tokens of math pretraining are wasted without chain-of-thought SFT.
The Instruct baseline nearly completely fails under weak-supervision RL, showing that generic instruction tuning does not transfer to reasoning generalization.

Other Notable Results

Under 70% label noise, Qwen still improves on MATH and SCIENCE, while Llama's training reward curves overlap across noise levels — it memorizes noise as readily as signal.
Majority-vote proxy rewards work only briefly on math-specialized models, with most runs collapsing by ~500 steps. Self-certainty rewards cause failure in all settings — classic reward hacking.
Without an external verifier, self-supervised RLVR remains a dangerous gamble.

Practical Takeaways

1. Use the saturation step as a diagnostic dashboard. If training reward plateaus with no downstream gains, stop training — more RL compute buys nothing. A very fast saturation (<100 steps) is a red flag pointing to insufficient SFT/pretraining, not a need for more data. 2. When weak supervision fails, invest compute upstream. Improve chain-of-thought coverage in SFT data and domain pretraining rather than extending RL. As the paper puts it: RL under weak supervision is best understood not as a training technique applied to a fixed model, but as the final stage of a pipeline whose success is largely determined before RL begins.

References

1. Shen, J., et al. (2026). *When Can LLMs Learn to Reason with Weak Supervision?* arXiv:2604.18574. 2. Guo, D., et al. (2025). DeepSeek-R1. arXiv:2501.12948. 3. Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300. 4. Yang, A., et al. (2024). Qwen2.5-Math Technical Report. arXiv:2409.12122. 5. Dubey, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. 6. He, C., et al. (2025). Skywork-OR1. arXiv:2504.16339. 7. Baker, B., et al. (2025). Monitoring Reasoning Models for Misbehavior. arXiv:2503.11926. 8. Burns, C., et al. (2023). Weak-to-Strong Generalization. arXiv:2312.09390. 9. Zhao, S., et al. (2025). Self-Certainty as a Training Signal for LLM Reasoning. 10. Zuo, Y., et al. (2025). Majority Voting as Reward Signal for LLM Reasoning. 11. Mahabadi, R., et al. (2025). Nemotron-CC. arXiv:2502.20967. 12. Guha, S., et al. (2025). OpenThoughts-114K. arXiv:2502.12059. 13. Wang, P., et al. (2025). Can LLMs Learn from a Single Example? 14. Shafayat, et al. (2025). Reward Hacking in Self-Supervised RLVR. 15. Sheng, G., et al. (2024). veRL: A Flexible and Efficient RL Framework for LLMs.