Not the Wrong Samples, the Wrong Intervention: From Reweighting to Rewriting in Training Data Attribution

Forum topic · ✨步子哥 · 2026-09-03

Summary

This post from zhichai.net discusses a paper by Yangsibo Huang et al. (Princeton/Google) arguing that influence functions (IF) for training data attribution are often wrongly blamed for weak downstream results. The real problem lies in the intervention method: reweighting influential training samples typically produces weak, inconsistent behavioral changes, because reweighting only amplifies existing signals and relies on a local linear approximation that fails at realistic weight scales. The paper proposes influence-guided response rewriting: instead of adjusting sample weights, the responses of IF-selected samples are replaced with behavior-aligned or behavior-opposed versions while keeping instructions intact. Experiments on epistemic abstention across four open-source LLMs show rewriting produces stronger, persistent, bidirectional behavior changes concentrated on the target behavior, while reweighting of the same samples has limited, unstable effects. IF-selected samples outperform random selection as rewriting targets, confirming IF correctly identifies behavior-relevant samples. Results also transfer to safety refusal, though stronger rewriting increases over-refusal. The post argues TDA evaluation must be intervention-aware: IF's accuracy should be tested with rewriting, not reweighting.

Imagine you are a chef trying to figure out which seasoning actually determines a dish's flavor.

Influence functions (IF) are the tool statistics gives you: they tell you "if I add a bit more of this seasoning, how much will the flavor change?" You use them to identify the key ingredients.

Then you run the experiment: double the amount of the key seasonings. Result — the dish barely changes.

You're confused: did the influence function compute the wrong answers, or is "doubling the amount" the wrong kind of intervention?

This paper answers exactly that question.

The Embarrassment of Influence Functions

Influence functions are a core tool of training data attribution (TDA). They estimate a training sample's "contribution" to a test sample by perturbing the training sample's weight and observing how the model's behavior on the test sample changes.

The logic is elegant: if a training sample strongly influences the test outcome, it is a "key sample."

But the problem arises at the validation step. When you actually use IF to find the "most influential" samples and then apply reweighting (adjusting their training weights) — the effect is often no better than picking random samples.

That's embarrassing. IF says "these samples matter," but when you follow its advice, nothing happens. Did IF get it wrong, or did you use it wrong?

The Core Insight

Yangsibo Huang et al. (Princeton + Google) draw a key distinction:

Influence functions did not misidentify samples — they correctly identified which samples carry behavior-relevant signal. The problem is the reweighting intervention itself.

Reweighting adjusts sample weights — effectively telling the model "look at this sample more often." But "seeing it more often" is not the same as "learning something different." If the sample's response itself does not carry the behavioral signal you want, showing it more just reinforces the existing signal; no new behavior emerges.

The paper's solution: influence-guided response rewriting. Use IF to find key samples, but don't adjust weights — replace the samples' responses. Keep the instruction, and swap the response for a "behavior-aligned" or "behavior-opposed" version.

Back to the Chef Analogy

Reweighting: IF says "salt matters," so you add more salt. But the amount is already sufficient — more just makes the dish saltier; it doesn't create a new flavor dimension.
Rewriting: IF says "salt matters," so you don't adjust the quantity — you switch to sea salt, rock salt, smoked salt. The same "salt" slot, but carrying different flavor signals.

Rewriting changes not "how many times the model sees it" but "what the model sees."

Experimental Design

The paper uses "epistemic abstention" (saying "I don't know" instead of guessing when uncertain) as the test scenario, with four behavior conditions:

Behavior-aligned: rewrite the responses of IF-selected samples to "correct abstention" versions
Behavior-opposed: rewrite them to "forcefully answer with the wrong answer"
Reweighting up: keep original responses, increase weights
Reweighting down: keep original responses, decrease weights

Tested on four open-source LLMs.

Results

Rewriting produces stronger, more persistent, bidirectional behavior change:

Behavior-aligned rewriting significantly increases abstention rates
Behavior-opposed rewriting significantly decreases abstention rates
Effects persist after training
Changes concentrate on the target behavior, without global drift

Reweighting the same samples: weak and inconsistent:

Reweighting up/down affects abstention rates far less than rewriting
Direction is unstable, sometimes even opposite to expectations
Even larger weight changes yield limited effects

IF-selected samples are indeed better rewriting targets than random ones — proving IF itself isn't wrong; it genuinely finds "behavior-relevant" samples. Reweighting just can't unlock their "intervention leverage."

Transfer to Safety

The paper also tests safety refusal with consistent conclusions: rewriting changes safety behavior more effectively than reweighting.

But there's a trade-off: stronger safety rewriting brings over-refusal — the model becomes overly conservative and refuses safe questions too. This relates to "judgment-gate decoupling": rewriting changes behavioral tendency but not judgment capability, so the model cannot distinguish "should refuse" from "shouldn't refuse."

Why Reweighting Falls Short

The paper's analysis points to a deeper reason:

Reweighting is a local linear approximation. Influence functions are derived from "infinitesimal weight perturbations" — they estimate how behavior changes when weights shift by a tiny amount. But real reweighting is not infinitesimal: you must at least double or halve weights to see an effect. At that scale, the local linear approximation breaks down.

Rewriting changes the signal a sample carries, not the signal's weight. This bypasses the local linear limitation — you don't need "more of the same signal"; you swap in a different signal.

This fits a broader conceptual lineage of "solving problems at a different layer": reweighting intervenes in weight space; rewriting intervenes in content space. When weight-space interventions fail because their linear assumption breaks, move to content space.

Implications for TDA Evaluation

The paper's title states the core: "intervention-aware evaluation of TDA methods."

Common practice today: use IF to find influential samples, then reweight and see the effect. If the effect is weak, conclude IF is inaccurate.

This paper says that evaluation logic has a structural flaw. IF may correctly identify "which samples carry behavioral signal," but reweighting cannot release that signal — you used the wrong intervention tool.

The correct evaluation should be: use IF to select samples, use rewriting as the intervention, then assess the effect. Only if rewriting also fails can you claim IF is inaccurate.

Conceptual Resonances

Another entry in the "solve at a different layer" lineage: reweighting (weight space) → rewriting (content space).
Another entry in the "evaluation blind spot" pattern: validating IF via reweighting is like measuring blood pressure with a thermometer — tool and target mismatched.
A variant of "judgment-gate decoupling": IF correctly identifies "which samples matter" (judgment), but reweighting cannot turn that judgment into actual behavior change (gate). Rewriting fixes the gate, not the judgment.
The "scalar illusion": managing "sample influence" as a scalar (more weight = more influence) ignores the direction and content dimensions of influence.

Practical Takeaways

For teams doing training data analysis and attribution:

1. Don't validate IF with reweighting alone. If reweighting underperforms, try rewriting before drawing conclusions. 2. IF-selected samples may genuinely matter — you just need a different intervention to unlock their value. 3. Rewriting can serve as a "behavior editing" tool: use IF to locate key samples, use rewriting to inject desired behavioral signals. Far more precise than reweighting.

---

Paper: arXiv:2609.02771 Authors: Yangsibo Huang, Da Yu, Xinyu Yang, Kang Guo, Bhuwan Dhingra, Peter Bartlett, Jacob Steinhardt, et al. (Princeton / Google / Berkeley) Code: influence functions computed with kronfluence