OP-Print: Tracing Membership Imprints in On-Policy Self-Distillation

Forum topic · 小凯 · 2026-10-25

Summary

On-policy self-distillation removes the need for an external teacher by using the student's own outputs as supervision, but its privacy risks have remained largely unexplored. Because two coupled views of the same training example are used, memorization may be amplified. A paper by Qi Li and Xinchao Wang (arXiv:2610.31616) presents the first study of membership inference attacks against on-policy self-distillation, introducing OP-Print, a unified attack framework. The key insight is that the supervision imprint in on-policy self-distillation is sparse and structured, concentrating on uncertainty-sensitive correction sites. In gray-box settings, OP-Print aggregates the privileged teacher-student residual correction signal along the same rollout; in black-box settings, it uses restart-based output-basin stability as a surrogate signal. Experiments show all baseline attacks perform no better than random guessing even with gray-box access, while OP-Print is widely validated as effective in both settings. The work highlights previously overlooked privacy leakage in self-distillation pipelines.

Paper Overview

Field: ML Authors: Qi Li, Xinchao Wang Published: 2026-10-22 arXiv: 2610.31616

Original Abstract

On-policy self-distillation removes the need for an external teacher by using the student's own outputs as supervision, yet its privacy risks remain largely unexplored. Using two coupled views of the same training example may strengthen its memorization. We present the first study of membership inference attacks against on-policy self-distillation, proposing OP-Print, a unified attack framework. OP-Print builds on our finding that the supervision imprint in on-policy self-distillation is sparse and structured, concentrating on uncertainty-sensitive correction sites. In gray-box settings, OP-Print aggregates the privileged teacher-student residual correction along the same rollout; in black-box settings, it measures restart-based output-basin stability as a surrogate. All baseline attacks random-guess even with gray-box access, while OP-Print is broadly validated in both settings.

Key Points

First membership inference attack study targeting on-policy self-distillation.
Supervision imprints are sparse and structured, concentrated at uncertainty-sensitive correction sites.
Gray-box: aggregates teacher-student residual correction signal along the same rollout.
Black-box: uses restart-based output-basin stability as a surrogate signal.
Baseline attacks perform at random-guess level even with gray-box access; OP-Print is effective in both settings.

--- *Auto-collected on 2026-10-26*