ModSleuth: Auditing Hidden Model Dependencies in Modern LLM Training Pipelines

Forum topic · 小凯 · 2026-06-12

Summary

Modern LLM training pipelines increasingly depend on other models for data generation, corpus filtering, output judging, and development decisions, creating recursive dependencies fragmented across heterogeneous public artifacts. This paper introduces ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. The authors identify the key challenge as defining what constitutes a dependency and reconciling artifact references across inconsistent documentation, rather than simple information extraction. They formalize direct versus indirect dependencies, represent heterogeneous pipeline roles through operation-centric relations, and resolve artifact identities across names, versions, and repositories. Applied to four artifact-rich public LLM releases, ModSleuth recovers 1,060 source-verified dependencies and builds large-scale dependency graphs that reveal multi-hop license obligations, training-evaluation coupling, discrepancies between release and training-time artifacts, and previously undiscoverable documentation inconsistencies. ModSleuth and the resulting dependency graphs are released publicly to support transparent analysis of the increasingly complex ecosystem underlying modern LLMs.

Paper Overview

Field: NLP Authors: Sanjay Adhikesaven, Haoxiang Sun, Sewon Min Published: 2026-06-10 arXiv: 2606.12385

Abstract

Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions. These dependencies are recursive: a model may depend on an upstream artifact whose own dependencies are documented only in separate releases and artifacts. As a result, the full dependency structure is fragmented across heterogeneous public artifacts, with complexity and recursive depth far outpacing humans' ability to trace.

The authors introduce ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. A key finding is that the primary challenge is no longer information extraction, but rather:

Defining what constitutes a dependency
Reconciling artifact references across inconsistent documentation

ModSleuth addresses these challenges by:

Formally distinguishing direct and indirect dependencies
Representing heterogeneous pipeline roles through operation-centric relations
Resolving artifact identities across names, versions, and repositories

Results

Applied to four public artifact-rich LLM releases, ModSleuth recovers 1,060 source-verified dependencies and constructs large-scale dependency graphs of modern LLM development. These graphs reveal:

Multi-hop license obligations
Training-evaluation coupling
Discrepancies between release-time and training-time artifacts
Documentation inconsistencies otherwise difficult to discover

ModSleuth and the resulting dependency graphs are publicly released to support transparent analysis of the increasingly complex ecosystem underlying modern LLMs.