RePAIR and Interactive Machine Unlearning: Teaching AI to Forget at Inference Time

Forum topic · 小凯 · 2026-04-15

Summary

RePAIR (Interactive Machine Unlearning through Prompt-Aware Model Repair) is a framework that lets users ask an AI model to forget specific information during conversation, rather than relying on the model provider to retrain. It uses three cooperating models: a Watchdog that detects forgetting intent and extracts the content to erase, a Surgeon that generates weight-repair instructions, and a Patient model that modifies its own parameters. The core technique, STAMP (Steering Through Activation Manipulation with PseudoInverse), computes a steering vector between activations of forget samples and refusal-like references, then uses a pseudoinverse-based weight update so forget inputs are pushed toward refusal while retain inputs stay unchanged. A low-rank variant, STAMP-LR, reduces complexity from O(d^3) to O(r^3 + r^2·d), giving roughly 3x speedup and enabling real-time, inference-time unlearning. In experiments on Llama-3-8B across harmful knowledge suppression, misinformation removal, and personal data erasure, STAMP-LR achieved forget accuracy of 0.00 while retaining over 73-84% retain accuracy, outperforming baselines like WGA and ASU. The approach shifts unlearning power from providers to users, with implications for privacy rights and AI safety, though limitations remain around retain-data storage, multimodal scaling, and the ethics of editable machine memory.

Overview

This post is a Feynman-style explainer of RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair (arXiv:2604.12820), a method that lets users command an LLM to permanently forget specific information via natural conversation — no retraining, no provider intervention.

Key points

The problem: LLMs absorb the entire internet during training, including harmful knowledge, misinformation, and private data. Retraining to remove anything is prohibitively expensive, and existing machine unlearning methods put all control in the hands of model service providers (MSPs).
IMU (Interactive Machine Unlearning): RePAIR shifts unlearning from batch processing at training time to real-time, user-triggered operations at inference time — like clearing browser history, but for AI memory.

The three-model architecture

| Role | Name | Responsibility | |------|------|----------------| | Detective | Watchdog Model | Monitors conversation, detects forgetting intent, extracts content to erase | | Surgeon | Surgeon Model | Generates repair code instructing the Patient how to modify its weights | | Patient | Patient Model | Autonomously modifies its own neural network parameters |

STAMP: the core technique

STAMP (Steering Through Activation Manipulation with PseudoInverse) works as follows:

1. The Watchdog extracts a *forget pair* (forget prompt p_f + forget response r_f). 2. Three datasets are built: forget set (D_f), retain set (D_r), and a reference set (D_ref) of natural refusal prompts (the base model tends to echo "I don't know"-style inputs, defining a refusal subspace). 3. A steering vector is computed: r_SV = (mean reference activations) − (mean forget activations). 4. A pseudoinverse-based solve finds new weights W_new = X^+ · O' (with regularization λI for stability) such that forget inputs' activations shift toward refusal while retain inputs remain unchanged. 5. STAMP-LR uses a low-rank approximation X ≈ A·B to reduce complexity from O(d³) to O(r³ + r²·d) — about 3x faster, making real-time unlearning feasible.

Experimental results (Llama-3-8B, harmful knowledge removal)

| Method | Acc_f ↓ | Acc_r ↑ | F-RL ↓ | R-RL ↑ | |--------|---------|---------|--------|--------| | WGA | 2.10 | 70.17 | 11.99 | 11.20 | | ASU | 0.90 | 68.39 | 7.91 | 12.13 | | STAMP | 0.00 | 70.13 | 6.55 | 7.13 | | STAMP-LR | 0.00 | 73.27 | 7.00 | 4.25 |

Perfect forgetting: Acc_f = 0.00 and F-RL = 0.00 on harmful knowledge — the model truly cannot recall the erased content.
Minimal collateral damage: retain accuracy stays above 73% (84.47% on misinformation removal), best among compared methods.
Three scenarios tested: harmful knowledge suppression, misinformation removal, and personal data erasure — state-of-the-art in all three.

Why it matters

Power shift to users: the "right to be forgotten" (already in GDPR) gains a practical technical implementation; users decide what the AI remembers.
AI safety: harmful knowledge can be patched out quickly by the community, like a digital immune system, instead of waiting weeks for provider retraining.
Open vs. closed balance: if open models can be user-corrected in real time, the risks of openness shrink.

Acknowledged limitations

Still requires a small retain buffer, raising on-device privacy compliance questions.
Multimodal settings remain resource-intensive.
Whether forgetting is ever truly irreversible in neural parameters is philosophically unresolved.
Editable AI memory raises ethical questions: historical revisionism, personalized truths, and potential abuse (e.g., erasing safety guardrails).

References

1. Rachapudi, J., et al. (2026). *RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair*. arXiv:2604.12820. https://arxiv.org/abs/2604.12820 2. Grattafiori, A., et al. (2024). *The Llama 3 Herd of Models*. arXiv:2407.21783 3. Wang, Z., et al. (2025). *Activation Steering for Harmful Knowledge Removal in LLMs* (WGA baseline) 4. Zade, A., et al. (2026). *ASU: Adversarial Similarity-based Unlearning* 5. Sun, Y., et al. (2024). *Test-Time Training for Distribution Adaptation in Vision*. NeurIPS 2024