RULER: Representation-Level Verification Metrics for Machine Unlearning

Forum topic · 小凯 · 2026-05-29

Summary

Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Existing verification protocols operate only at the output level—via membership inference attacks, retain accuracy, and forget-set accuracy—but a model can pass all three checks while still encoding forgotten records in its intermediate representations. RULER introduces representation-level verification metrics: the oracle-comparative metric M2 measures whether forget-set records occupy the same representational position as in a model retrained without them, and the oracle-free metric M4 detects residual traces from the unlearned model's internal similarity structure alone, requiring no retraining. Experiments show that four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model, M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect sizes growing as the proportion of forgotten data increases. M4 serves as a pre-unlearning diagnostic across tabular data, images, clinical text, and face recognition, detecting identity-level memorization signals that existing methods cannot fully erase.

Paper Overview

Field: Machine Learning Authors: Georgina Cosma, Axel Finke Published: 2026-05-28 arXiv: 2605.27569

Abstract

Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current protocols verify this at the output level through membership inference, retain accuracy, and forget-set accuracy, but a model can satisfy all three whilst still encoding forgotten records in its intermediate representations.

The authors introduce RULER, a set of representation-level verification metrics:

M2 (oracle-comparative): measures whether forget-set records occupy the same representational position as in a model retrained without them.
M4 (oracle-free): detects residual traces from the unlearned model's internal similarity structure alone, without requiring retraining.

Key Findings

Four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model, M2 detects significant residuals in 10 of 12 conditions (p<0.05).
Effect sizes grow as the proportion of forgotten data increases.
M4 works as a pre-unlearning diagnostic tool across multiple domains—tabular data, images, clinical text, and face recognition—and detects identity-level memorization signals that existing unlearning methods cannot fully erase.

Why It Matters

RULER reveals that passing standard output-level unlearning audits is insufficient, and offers both a rigorous (M2) and a practical, retraining-free (M4) way to verify whether forgotten records truly disappear from a model's internal representations.

--- *Auto-collected on 2026-05-29.*