Not the Wrong Samples, the Wrong Intervention: From Reweighting to Rewriting in Training Data Attribution

Imagine you are a chef trying to figure out which seasoning really determines a dish's flavor.

The influence function is the tool statistics gives you: it tells you "how much would the flavor change if you added a bit more of this seasoning." You use it to identify the key seasonings.

Then you run an experiment: you double the amount of those key seasonings. The result — the dish tastes almost exactly the same.

You are confused: did the influence function get it wrong, or is "doubling the amount" itself a flawed intervention?

That is exactly the question this paper answers.

The Embarrassment of Influence Functions

The influence function (IF) is a core tool for training data attribution (TDA). It perturbs the weight of a single training sample and observes how the model's behavior on a test sample changes, thereby estimating that sample's "contribution."

The logic is elegant: if a training sample heavily influences a test outcome, it is a "key sample."

The problem appears at the validation step. When you actually use IF to find the "most influential" samples and then reweight them, the effect is often no better than randomly picking a batch of samples.

Awkward. IF says "these samples matter," but when you follow its advice, nothing happens. Is IF wrong, or are you using it wrong?

The Core Insight

Yangsibo Huang et al. (Princeton + Google) make a key distinction:

The influence function did not misidentify samples — it correctly identified which samples carry behavior-relevant signal. The problem is the reweighting intervention itself.

Reweighting adjusts a sample's weight — essentially telling the model "look at this sample more often." But seeing a sample more often is not the same as learning something different. If the sample's response does not carry the behavior signal you want, repetition only reinforces existing signal; it produces no new behavior change.

The paper's solution: influence-guided response rewriting. Use IF to find key samples, but instead of adjusting weights, replace the responses. Keep the instruction and swap the response for a behavior-aligned or behavior-opposed version.

The Chef Analogy

• Reweighting: IF says "salt matters," so you add more salt. But the salt is already sufficient — more just makes it saltier, it doesn't create a new flavor dimension.
• Rewriting: IF says "salt matters," so you don't change the amount — you swap in sea salt, rock salt, smoked salt. The same "salt" slot, but carrying different flavor signals.

Rewriting changes not "how many times you see it" but "what you see."

Experimental Design

The paper uses epistemic abstention — the model should say "I don't know" when uncertain rather than guessing — as the test scenario.

Four behavior conditions:

• Behavior-aligned: rewrite responses of IF-selected samples to "correct abstention"
• Behavior-opposed: rewrite them to "forcefully answer incorrectly"
• Reweighting up: keep original responses, increase weight
• Reweighting down: keep original responses, decrease weight

Tested on four open-source LLMs.

Results

Rewriting produces stronger, more persistent, bidirectional behavior change:

• Behavior-aligned rewriting significantly increases abstention rates
• Behavior-opposed rewriting significantly decreases abstention rates
• Effects persist after training
• Changes concentrate on the target behavior without global drift

Reweighting the same samples: weak and inconsistent:

• Reweighting up/down affects abstention far less than rewriting
• Direction is unstable, sometimes opposite to expectations
• Even larger weight changes yield limited effects

IF-selected samples are indeed better rewriting targets than random selection — proving IF itself is not wrong; it finds behavior-relevant samples. Reweighting simply cannot unlock their "intervention leverage."

Transfer to Safety

The paper also tests safety refusal with consistent conclusions: rewriting changes safety behavior more effectively than reweighting.

But there is a trade-off: stronger safety rewriting brings over-refusal — the model becomes overly conservative and refuses safe questions too. This relates to the "judgment-gate decoupling": rewriting changes behavioral tendency without changing judgment capability, so the model cannot distinguish "should refuse" from "should not refuse."

Why Reweighting Is Not Enough

Reweighting is a local linear approximation. Influence functions are derived from infinitesimal weight perturbations — they estimate behavior change for tiny weight shifts. But real reweighting is not infinitesimal; you need at least a doubling or halving to see any effect. At that scale, the local linear approximation fails.

Rewriting changes the signal a sample carries, not its weight. This bypasses the linear approximation limitation — you don't need "more of the same signal," you directly swap in a different signal.

This fits a broader conceptual lineage: reweighting intervenes in weight space, rewriting intervenes in content space. When weight-space interventions fail due to broken assumptions, move to content space.

Implications for TDA Evaluation

The paper's title makes the point: "intervention-aware evaluation of TDA methods."

The common TDA evaluation recipe: use IF to find influential samples, reweight them, and judge IF by the outcome. If the effect is weak, conclude IF is inaccurate.

This paper says that logic has a structural flaw. IF may have correctly identified samples carrying behavioral signal, but reweighting cannot release that signal — you used the wrong intervention tool.

The correct evaluation should be: select samples with IF, intervene via rewriting, then judge. Only if rewriting also fails can you say IF is inaccurate.

Practical Takeaways

For teams doing training data analysis and attribution:

1. Don't validate IF with reweighting alone. If reweighting shows weak effects, try rewriting before drawing conclusions. 2. IF-selected samples may genuinely matter — you just need a different intervention to release their value. 3. Rewriting works as a "behavior editing" tool: use IF to locate key samples, use rewriting to inject desired behavioral signals — far more precise than reweighting.

---

Paper: arXiv:2609.02771 Authors: Yangsibo Huang, Da Yu, Xinyu Yang, Kang Guo, Bhuwan Dhingra, Peter Bartlett, Jacob Steinhardt, et al. (Princeton / Google / Berkeley) Code: influence functions computed with kronfluence