Design Docs Are All You Need: SMART, an AI-Native ML Performance Modeling Library Built from Natural-Language Design Docs

Forum topic · 小凯 · 2026-09-09

Summary

A Chinese tech forum post summarizes the arXiv paper 'Design Docs Are All You Need' (arXiv:2609.05364) by Samuel Kushnir et al., which introduces SMART, a rigorous symbolic performance-modeling library for ML systems. Because ML performance models are repeatedly invalidated by new models and hardware, the authors argue that regenerating an entire library with AI coding agents is cheaper than incrementally patching tech debt. SMART's main branch contains almost no code: the repository is a DAG of self-contained natural-language design docs, and coding sub-agents regenerate implementations from the docs on each version update, with every human change made as a natural-language doc edit, making the system self-documenting. Two elements make regeneration reliable: a doc style built around step-by-step worked examples that serve as contextual demonstrations, and a minimal recursively defined operation IR with SymPy symbolic cost expressions, a fast analytical summary mode for large sweeps, and a slow module-scheduling mode for fine-grained studies. Regenerated implementations reproduce hand-audited reference models—including DeepSeek-V3 serving on TPU pod slices—to rounding precision, suggesting design docs, not code, can be the durable artifact for ML system co-design.

Paper Overview

Field: Machine Learning Authors: Samuel Kushnir, Kimia Noorbakhsh, Kavya Sreedhar, Liqun Cheng, Ming Liu, Parthasarathy Ranganathan, Mohammad Alizadeh, Fred Kjolstad, Suvinay Subramanian Published: 2026-09-04 arXiv: 2609.05364

Abstract (Translation)

Machine-learning performance modeling is a uniquely hostile terrain for long-lived software: the assumptions baked into today's abstractions are invalidated by tomorrow's models and systems, forcing perpetual refactoring of performance-modeling frameworks. Meanwhile, AI coding agents have become fast and capable enough that regenerating an entire library is cheaper than paying down the tech debt of incrementally patching it.

This paper describes SMART, a rigorous symbolic performance-modeling library for ML systems whose main branch contains almost no code: the repository is a DAG of self-contained natural-language design docs, coding sub-agents regenerate the implementation from only the docs on new version updates, and every human change is a natural-language edit to a doc—making the system self-documenting.

Two elements make regeneration reliable:

1. A design-doc style built around step-by-step worked examples, which act as contextual demonstrations for the generating agents. 2. A minimal, recursively defined operation IR with symbolic (SymPy) cost expressions, a fast analytical summary mode for large-scale sweeps, and a slow module-scheduling mode for fine-grained scheduling studies.

Regenerated implementations reproduce hand-audited reference models—including DeepSeek-V3 serving on TPU pod slices—to rounding precision, indicating that design docs, rather than code, can be the durable artifact for ML system co-design tools.

Key Takeaways

Docs as the source of truth: the codebase is regenerated from a DAG of natural-language design documents rather than incrementally maintained.
AI agents as compilers: coding sub-agents turn docs into working implementations on every version update.
Reliability mechanisms: worked examples plus a symbolic operation IR (SymPy-based cost models) make regeneration reproducible.
Validation: regenerated models match hand-audited references, including DeepSeek-V3 serving on TPU pod slices, to rounding precision.