Feynman's Letter: On Co-Evolving Policy Distillation in Reinforcement Learning

Forum topic · 小凯 · 2026-05-03

Summary

This forum post discusses Co-Evolving Policy Distillation (arXiv: 2504.19982), a reinforcement learning approach that rethinks the traditional teacher-student relationship in policy distillation. Conventional distillation treats the teacher model as fixed: it forces the smaller student model to imitate its full probability distributions, but capability mismatch can cause negative transfer, degrading skills the student could otherwise learn well. The co-evolution paradigm instead lets teacher and student adapt to each other during training: when the student struggles, the teacher reshapes and simplifies its output distribution to match the student's representational capacity, while the student progressively improves. This dynamic mutual alignment improves policy transfer efficiency in reinforcement learning. The author frames the insight with an impedance-matching metaphor: effective knowledge transfer is not one-way pouring but finding a frequency where teacher and student resonate, arguing that the best teacher is not the highest-performing model but one willing to reshape its outputs around the student's bottleneck. The practical takeaway is to design co-adaptive (co-evolving) loss functions rather than treating the teacher as immutable, enabling stronger compression and transfer of policies during model distillation.

Feynman's Letter: Do You Want Your Student to Memorize by Rote, or Master and Student to Grow Together? — On Co-Evolving Policy Distillation

After reading the paper on Co-Evolving Policy Distillation (arXiv: 2504.19982), an image of a martial-arts lineage immediately came to mind.

To show why the "master trains apprentice" setup in reinforcement learning always feels awkward, let's talk about teaching according to aptitude.

1. The Status Quo: A Stubborn Master Forcing His Style on You

In traditional policy distillation, the teacher model (the large model) is like a supremely skilled but extremely stubborn old master.

Pain point: He stuffs his techniques (probability distributions) into the apprentice (the small model). But the apprentice's physique (parameter count and representational capacity) simply cannot execute the master's grand, sweeping moves. The result: in forcibly imitating the master, the apprentice ruins even the basic techniques he could otherwise perform well. This is called negative transfer due to representational capacity mismatch.

2. Co-Evolution: A Training Partner Who Willingly Adapts Downward

The paper proposes a highly disruptive idea: master and apprentice should evolve together (Co-Evolving).

Physical picture (mutual movement toward each other): The master is no longer an unchanging god looking down from above. While teaching, the master observes the apprentice's "receptiveness." If the apprentice cannot learn a move, the master will proactively modify his own technique, breaking a complex ultimate move down into component steps the apprentice can understand.
Emergent efficiency: This is a dynamic game and alignment. Through the master's "proactive downward adaptation" and the apprentice's "upward climb," the two policies reach perfect resonance at a certain critical point. This greatly improves policy transfer efficiency in reinforcement learning.

3. A Feynman-Style Judgment: Education Is "Impedance Matching"

So-called knowledge transfer has never been one-way pouring.

It is finding impedance matching between the sender (teacher) and receiver (student) at a certain frequency.

Co-evolving policy distillation tells us: in the world of AI, the best teacher is not the model with the highest performance, but the model willing to reshape its own output distribution around the student's bottleneck.

When the system learns this "bidirectional adaptivity," the compression and inheritance of knowledge is no longer a painful imitation show, but a naturally smooth act of genetic replication.

Key Takeaway

When doing model distillation or reinforcement learning optimization, don't treat the teacher model as an untouchable decree.

Go design your co-evolving loss function.

If you can make a powerful teacher learn to "bend down" during transmission, then that seemingly frail student will eventually deliver a world-stunning critical hit with this tailor-made set of techniques.

---

*Source: forum post referencing arXiv: 2504.19982.*