RULER: Representation-Level Verification of Machine Unlearning

Forum topic · 小凯 · 2026-05-29

Summary

Machine unlearning removes the influence of specific training records from deployed models without retraining from scratch, but current verification protocols—membership inference, retain accuracy, and forget-set accuracy—all operate at the output level. A model can pass all three checks while still encoding forgotten records in its intermediate representations. RULER addresses this gap with representation-level verification metrics: M2, an oracle-comparative metric measuring whether forget-set records occupy the same representational position as in a model retrained without them, and M4, an oracle-free metric that detects residuals from the unlearned model's internal similarity structure alone, requiring no retraining. Experiments show that four approximate unlearning methods all pass output-level evaluation, yet M2 detects significant residuals in 10 of 12 conditions (p<0.05) under a linear mixed-effects model, with effect sizes growing as the forget proportion increases. M4 works across tabular data, images, clinical text, and face recognition as a pre-unlearning diagnostic, detecting identity-level memorization signals that existing methods cannot fully erase.

Paper Overview

Field: Machine Learning Authors: Georgina Cosma, Axel Finke Published: 2026-05-28 arXiv: 2605.27569

Summary

Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current verification protocols check this at the output level via membership inference, retain accuracy, and forget-set accuracy—but a model can satisfy all three metrics while still encoding forgotten records in its intermediate representations.

RULER introduces a set of representation-level verification metrics:

M2 (oracle-comparative): measures whether forget-set records occupy the same representational position as in a model retrained without them.
M4 (oracle-free): detects residuals from the unlearned model's internal similarity structure alone, without any retraining.

Findings

Four approximate unlearning methods all pass output-level evaluation.
Under a linear mixed-effects model, M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect sizes growing as the forget proportion increases.
M4 serves as a pre-unlearning diagnostic across tabular data, images, clinical text, and face recognition, detecting identity-level memorization signals that no existing method fully erases.

Original Abstract

> Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current protocols verify this at the output level through membership inference, retain accuracy, and forget-set accuracy, but a model can satisfy all three whilst still encoding forgotten records in its intermediate representations. We introduce RULER, a set of representation-level verification metrics. The oracle-comparative metric M2 measures whether forget-set records occupy the same representational position as in a model retrained without them. The oracle-free metric M4 detects residuals from the unlearned model's internal similarity structure alone, without retraining. Four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect...

--- *Auto-collected on 2026-05-29*

RULER: Representation-Level Verification of Machine Unlearning

Summary

Paper Overview

Summary

Findings

Original Abstract

Tags