Influcoder: Distilling Decoders' Gradient Influence Rankings for Scalable Data Attribution

Forum topic · 小凯 · 2026-06-14

Summary

Influcoder (arXiv:2606.13668) is a fast, cost-effective method for influence-based Data Attribution (DA) in large language models. As LLM capabilities grow, curating high-quality training datasets by filtering samples has become increasingly important, and DA methods estimate how individual training samples precondition a model to generate specific outputs. Traditional approaches based on the influence-function paradigm quantify this relationship but fall short in processing speed and storage compactness, making them impractical to deploy on large datasets. Influcoder addresses this by distilling decoders' gradient influence rankings, offering an efficient and scalable alternative for dataset curation. Authored by Dimitri Kachler, Damien Sileo, and Pascal Denis in the NLP field, the paper was published on arXiv in June 2026. This post summarizes the paper's motivation, approach, and contributions for readers interested in LLM training data attribution.

Paper Overview

Field: NLP Authors: Dimitri Kachler, Damien Sileo, Pascal Denis Published: 2026-06-11 arXiv: 2606.13668

Abstract

With the growth of LLMs' capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. Data Attribution (DA) methods aim to estimate how individual samples precondition a model to generate certain outputs. Many methods quantify this through influence functions, but they lack processing speed and storage compactness for large datasets.

Key Contributions

Problem: Influence-function-based Data Attribution methods are too slow and storage-intensive for practical deployment at scale on large training corpora.
Proposed solution: Influcoder, a quick and cost-effective approach to influence-based Data Attribution at scale.
Idea: Distill decoders' gradient influence rankings into an efficient model, enabling rapid estimation of how training samples influence generated outputs.