Not the Wrong Samples, the Wrong Intervention: From Reweighting to Rewriting in Training Data Attribution

This post analyzes a recent paper on training data attribution (TDA) and influence functions. The core question: when influence-function-guided reweighting fails to change model behavior, is the influence function wrong—or is the intervention itself the problem? The paper's answer: the attribution is fine; the intervention method is wrong.

The Embarrassment of Influence Functions

Influence functions (IF) are a core tool of training data attribution. They estimate a training sample's contribution to a test sample's behavior by perturbing the training sample's weight and observing the change in model output.

The logic is elegant: if a training sample strongly influences the test result, it is a "key sample."

But validation is where things break down. When you use IF to find the "most influential" samples and then apply reweighting to them, the effect is often no better than randomly chosen samples.

IF says "these samples matter," but following its advice doesn't work. Is IF computed incorrectly, or is it being used incorrectly?

The Core Insight

Yangsibo Huang et al. (Princeton + Google) draw a key distinction:

Influence functions do not misidentify samples—they correctly identify which samples carry behavior-relevant signal. The problem is the "reweighting" intervention itself.

Reweighting adjusts sample weights—telling the model "look at this sample more often." But looking more often does not mean learning something different. If the sample's response itself does not carry the desired behavioral signal, seeing it more only reinforces the existing signal and produces no new behavior change.

The paper's solution: influence-guided response rewriting. Use IF to find key samples, but instead of adjusting weights, replace the responses. Keep the instruction and swap the response for a "behavior-aligned" or "behavior-opposed" version.

The Chef Analogy

Back to the chef:

• Reweighting: IF says "salt matters," so you add more salt. But the amount is already sufficient—more salt just makes the dish saltier; it doesn't create a new flavor dimension.
• Rewriting: IF says "salt matters," so you don't change the amount—you switch to sea salt, rock salt, smoked salt. Same "salt" position, but carrying different flavor signals.

Rewriting changes not "how many times you see it," but "what you see."

Experimental Design

The paper uses "epistemic abstention" as the test scenario: the model should say "I don't know" when uncertain rather than guess.

Four behavioral conditions:

• Behavior-aligned: rewrite IF-selected samples' responses to "correct abstention"
• Behavior-opposed: rewrite to "forcefully answer with the wrong answer"
• Reweighting up: keep original responses, increase weights
• Reweighting down: keep original responses, decrease weights

Tested on four open-source LLMs.

Results

Rewriting produces stronger, more persistent, bidirectional behavior change:

• Behavior-aligned rewriting significantly increases abstention rates
• Behavior-opposed rewriting significantly decreases abstention rates
• The effect persists after training
• Changes concentrate on the target behavior without global drift

Reweighting the same samples: weak and inconsistent:

• Reweighting up/down affects abstention rates far less than rewriting
• The direction is unstable, sometimes opposite to expectation
• Even larger weight changes yield limited effects

IF-selected samples are indeed better than random selection as rewriting targets—proving IF itself is not wrong: it genuinely finds "behavior-relevant" samples. Reweighting just cannot unleash their "intervention leverage."

Transfer to Safety Scenarios

The paper also tests safety refusal. The conclusion holds: rewriting changes safety behavior more effectively than reweighting.

But there is a trade-off: stronger safety rewriting brings over-refusal—the model becomes overly conservative and refuses safe questions too. This connects to "judgment-gate decoupling": rewriting changes behavioral tendency but not judgment ability, so the model cannot distinguish "should refuse" from "should not refuse."

Why Reweighting Isn't Enough

The paper's analysis points to a deeper cause:

Reweighting is a local linear approximation. Influence functions are derived from "infinitesimal weight perturbations"—they estimate how behavior changes when weights shift slightly. But practical reweighting is not infinitesimal: you need at least to double or halve weights to see an effect. At that scale, the local linear approximation fails.

Rewriting changes the signal a sample carries, not the signal's weight. This bypasses the local linear approximation—you don't need "more of the same signal"; you directly swap in a different signal.

This fits the "change the level to solve the problem" lineage: reweighting intervenes in weight space, rewriting intervenes in content space. When weight-space intervention fails due to broken linear assumptions, move to content space.

Implications for TDA Evaluation

The paper's title says it: "intervention-aware evaluation of TDA methods."

Common current practice: use IF to find influential samples, apply reweighting, check the effect. If the effect is poor, conclude IF is inaccurate.

This paper argues that evaluation logic has a structural flaw. IF may correctly identify "which samples carry behavioral signal," but reweighting cannot release that signal—you used the wrong intervention tool.

The correct evaluation: use IF to select samples, use rewriting as the intervention, then check the effect. Only if rewriting also fails can you say IF is inaccurate.

Conceptual Lineage

This paper resonates with several existing concepts:

• "Change the level to solve the problem": reweighting (weight space) → rewriting (content space). When interventions at the old level fail due to broken assumptions, switch levels.
• "Evaluation blind spot law": TDA evaluation validates IF via reweighting—like measuring blood pressure with a thermometer. Tool and target don't match.
• "Judgment-gate decoupling" variant: IF correctly judges "which samples matter," but reweighting cannot turn that judgment into actual behavior change. Rewriting fixes the gate, not the judgment.
• "Scalar illusion": treating "sample influence" as a scalar (bigger weight = bigger influence) ignores the "direction" and "content" dimensions of influence.

Practical Takeaways

For teams doing training data analysis and attribution:

1. Don't validate IF with reweighting alone. If reweighting underperforms, try rewriting before drawing conclusions. 2. IF-selected samples may genuinely matter—you just need a different intervention to unlock their value. 3. Rewriting can serve as a "behavior editing" tool: use IF to locate key samples, use rewriting to inject desired behavioral signals. Far more precise than reweighting.

---

Paper: arXiv:2609.02771 Authors: Yangsibo Huang, Da Yu, Xinyu Yang, Kang Guo, Bhuwan Dhingra, Peter Bartlett, Jacob Steinhardt, et al. (Princeton / Google / Berkeley) Code: influence functions computed with kronfluence