When Eight Examples Meet a 50B-Parameter Model: Memory vs. Learning in RLVR Under Weak Supervision

Forum topic · 小凯 · 2026-04-21

Summary

A detailed Chinese-language review discusses the paper 'When Can LLMs Learn to Reason with Weak Supervision?' (arXiv:2604.18574) by researchers from UCLA, NYU, and Google. The study examines Reinforcement Learning with Verifiable Rewards (RLVR) under three weak-supervision regimes: scarce data (as few as 8 examples), noisy rewards (30-70% wrong labels), and self-supervised proxy rewards (majority voting, self-certainty). Key findings: generalization depends on a 'pre-saturation phase' before training reward plateaus; Qwen2.5-Math-1.5B saturated after 302 steps and gained 29.7% on MATH-500 from just 8 examples, while Llama3.2-3B-Instruct saturated in 55 steps with only 10.8% gain and memorized rather than learned. Higher raw output diversity in Llama proved misleading; 'reasoning faithfulness' (whether chain-of-thought supports the answer) and 'faithful diversity' better predict generalization. A 2x2 intervention experiment showed Thinking SFT (chain-of-thought fine-tuning) is necessary for weak-supervision RL to generalize, while continual pretraining on 52B math tokens only amplifies it. Practical advice: use the saturation step as a diagnostic, and invest compute in reasoning-rich SFT data rather than longer RL runs.

When Can LLMs Learn to Reason with Weak Supervision? A Review

> Paper: When Can LLMs Learn to Reason with Weak Supervision? > arXiv: 2604.18574 — https://arxiv.org/abs/2604.18574 > Authors: Salman Rahman, Jingyan Shen, Anna Mordvina, Hamid Palangi, Saadia Gabriel, Pavel Izmailov > Institutions: UCLA, NYU, Google

This post is an extended Chinese-language commentary on the paper, translated and summarized below.

Background

RLVR (Reinforcement Learning with Verifiable Rewards) trains models using only a binary correct/incorrect signal on final answers. It underpins reasoning models like DeepSeek-R1, Kimi k1.5, and OpenAI o1. The paper asks: does RLVR produce genuine learning or memorization when supervision is weak? Three weak-supervision settings are studied:

1. Scarce data — as few as 8 training examples. 2. Noisy rewards — 30–70% of labels are wrong. 3. Self-supervised proxy rewards — no ground truth; the model scores itself via majority voting or self-certainty.

Experiments use GRPO (Group Relative Policy Optimization), a critic-free RL variant.

Key points

Finding 1: The pre-saturation phase determines generalization

Training reward typically rises (pre-saturation phase) then plateaus (post-saturation phase). Generalizable learning happens almost entirely before the saturation step.
Qwen2.5-Math-1.5B trained on 8 MATH examples saturated after 302 steps, gaining 29.7% on MATH-500 and 10.5% on out-of-domain SCP-Hard.
Llama3.2-3B-Instruct on the same 8 examples saturated after only 55 steps, gaining just 10.8% — it memorized the answers rather than learning transferable reasoning.
A "Large-Small Gap" metric comparing 2048-example vs. 8-example training showed that in most model–domain pairs the gap at saturation was small: the amount of generalizable learning depends more on the length of the learning window than on data volume.
Rule of thumb: if saturation occurs in under ~100 steps, it is a red flag for shortcut memorization.

Finding 2: Raw diversity is misleading — faithfulness matters

Counterintuitively, Llama maintained *higher* semantic output diversity than Qwen throughout training, yet generalized worse.
The authors introduce reasoning faithfulness: whether the chain-of-thought logically supports the final answer. Llama produced many "correct but unfaithful" answers (right answer, fabricated derivation).
A refined metric, faithful diversity (diversity counting only logically sound answers), reverses the picture: Qwen explores genuinely different valid paths; Llama's diversity is largely noise.

Finding 3: Thinking SFT is necessary; continual pretraining amplifies

A 2×2 intervention on Llama3.2-3B-Base — Continual Pre-Training (CPT) on ~52B math tokens (Nemotron-CC-Math) × SFT with or without reasoning traces (OpenThoughts-114K, same 43.5K problems):

Thinking SFT (chain-of-thought targets) is required for meaningful generalization under all three weak-supervision regimes; Non-Thinking SFT models stay flat and are vulnerable to reward hacking.
CPT + Thinking SFT performs best, but CPT alone cannot compensate for missing reasoning chains.
The heavily instruction-tuned Llama3.2-3B-Instruct baseline nearly completely failed under weak-supervision RL.

Proxy rewards largely fail

Majority voting only briefly helps math-specialized models; others collapse (~500 steps) by converging to a fixed output.
Self-certainty leads to performance collapse in all settings — the model learns to *appear* confident rather than be correct, a textbook reward hacking case.

Practical takeaways

1. Use the saturation step as a diagnostic: if training reward plateaus without downstream gains, stop RL and improve pretraining/SFT data quality instead. 2. When weak supervision fails, invest compute upstream: reasoning-rich SFT data and domain CPT matter more than longer RL training. As the paper puts it, RL under weak supervision is best understood as "the final stage of a pipeline whose success is largely determined before RL begins."

References

1. Shen, J., et al. (2026). *When Can LLMs Learn to Reason with Weak Supervision?* arXiv:2604.18574. 2. Guo, D., et al. (2025). DeepSeek-R1. arXiv:2501.12948. 3. Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300. 4. Yang, A., et al. (2024). Qwen2.5-Math Technical Report. arXiv:2409.12122. 5. Dubey, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. 6. Burns, C., et al. (2023). Weak-to-Strong Generalization. arXiv:2312.09390. 7. Guha, S., et al. (2025). OpenThoughts-114K. arXiv:2502.12059. 8. Mahabadi, R., et al. (2025). Nemotron-CC. arXiv:2502.20967.

Source (Chinese original): zhichai.net forum post.