ModSleuth: Auditing Invisible Dependencies in Modern LLM Training Pipelines

Forum topic · 小凯 · 2026-06-12

Summary

Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions, creating recursive dependencies whose full structure is fragmented across heterogeneous public artifacts. This arXiv paper (2606.12385) by Sanjay Adhikesaven, Haoxiang Sun, and Sewon Min introduces ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. The key challenges are defining what constitutes a dependency and reconciling artifact references across inconsistent documentation. ModSleuth distinguishes direct from indirect dependencies, represents heterogeneous pipeline roles via operation-centric relations, and resolves artifact identities across names, versions, and repositories. Applied to four artifact-rich LLM releases, it recovers 1,060 source-verified dependencies and builds large-scale dependency graphs that reveal multi-hop license obligations, training-evaluation coupling, discrepancies between release and training-time artifacts, and documentation inconsistencies. ModSleuth and the resulting dependency graphs are released to support transparent analysis of the ecosystem underlying modern LLMs.

Overview

Field: NLP Authors: Sanjay Adhikesaven, Haoxiang Sun, Sewon Min Released: 2026-06-10 arXiv: 2606.12385

Abstract (translated)

Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions. These dependencies are recursive: a model may depend on an upstream artifact whose own dependencies are documented only in separate releases and artifacts. As a result, the full dependency structure is fragmented across heterogeneous public artifacts, with complexity and recursive depth far outpacing humans' ability to trace.

The authors introduce ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. They find that the primary challenge is no longer information extraction, but defining what constitutes a dependency and reconciling artifact references across inconsistent documentation. The paper formalizes solutions to these challenges by:

Distinguishing direct and indirect dependencies
Representing heterogeneous pipeline roles via operation-centric relations
Resolving artifact identities across names, versions, and repositories

Findings

Applying ModSleuth to four artifact-rich LLM releases, the system recovers 1,060 source-verified dependencies and constructs large-scale dependency graphs of modern LLM development. These graphs reveal:

Multi-hop license obligations
Training-evaluation coupling
Discrepancies between release-time and training-time artifacts
Documentation inconsistencies that would otherwise be hard to discover

ModSleuth and the resulting dependency graphs are released to support transparent analysis of the increasingly complex ecosystem underlying modern LLMs.

--- *Auto-collected on 2026-06-12*