> TL;DR: The paper's solid contribution is a clean experiment showing that naive uniform LoRA falls far short of full fine-tuning on multi-step procedures; its overreach is elevating this empirical negative result into a theorem-like claim that "procedural knowledge is fundamentally not low-rank." We should trust the behavioral findings, and be cautious with the headline claim.
1. What This Paper Is
- Title: *Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures*
- Authors: Simon Dennis, Kevin Shabahang, Hao Guo, Rivaan Patil (University of Melbourne)
- Identifier: arXiv:2607.21612 (v1, 14 pages, submitted 2026-07)
- Core question: LoRA-style parameter-efficient fine-tuning (PEFT) has become the default fine-tuning method, repeatedly validated on instruction following, style transfer, and factual adaptation. But does LoRA match full fine-tuning for *procedural knowledge* — the ability to traverse conditional branches through multiple steps to a terminal state?
- Full fine-tuning is like rewriting an entire cookbook — every step and every heat level written into muscle memory.
- LoRA only adds two lines of margin notes (ΔW = B·A with tiny rank r), hoping those notes can change the flavor of the whole book.
- Instruction following and style transfer are "local, smooth" flavors — margin notes suffice.
- Procedural knowledge is a dance of "choose a branch, decide the next step from prior context, loop until the end" — it spans the whole book and is interdependent; two margin notes can't capture it.
- Completion rate measures the surface token distribution; task success measures the state→action mapping — the two are decoupled. Low-rank B·A suffices to memorize frequent n-grams and template utterances, but struggles to encode conditional transitions in a latent state space (effective rank 888+).
- Root of high-rank degradation: decision trees are sparse, local, and conditional (86–2381 paths, nested loops); low-rank spaces only express smooth, globally correlated perturbations. As r grows, capacity is wasted fitting specific phrasings of particular paths — overfitting the surface rather than abstract transition rules.
- The MLP is the real bottleneck: the state→action mapping lives in the MLP, which has the highest effective rank (down_proj 1872) and largest capacity gap; K/V appearing "adaptable" is purely because GQA compresses their maximum rank to a few hundred — a dimensional artifact, not evidence that K/V carries procedural knowledge.
- Only uniform-rank naive LoRA was tested; heterogeneous-rank (AdaLoRA-style) variants were not.
- The title's "not low-rank / fundamental" claim exceeds the evidence. A load-bearing assumption: that full FT's ΔW is the "correct solution" LoRA must approximate. Eckart–Young only bounds the error *against that particular ΔW* — it does not prove that no low-rank update exists that can reproduce the behavior. The paper's Limitations section admits this.
- Free/hidden parameters: α=2r, LR 2e-4 vs 2e-5 (LoRA effectively gets 10× and wasn't tuned per-rank), and the effective-rank threshold is never defined.
- LLM judges with no human verification; only the Qwen family; only three customer-service domains.
- Untested variants: AdaLoRA/heterogeneous rank, LoRA+, DoRA, VeRA, QLoRA, per-rank tuning — any one matching full FT at r≤128 would refute the "structural infeasibility" claim.
- The "surface fluency" trap: mainstream agents today (LoRA-based base + external prompt/tool orchestration) delegate procedural correctness to prompts and schedulers; anything requiring in-model internalization of conditional branches and cross-turn state will go astray when LoRA's rank cap of 128 captures only 43–51% of the energy. Failure modes map one-to-one: hallucinated steps ↔ overfit conversation patterns; lost state ↔ late-turn state-loss rebound; infinite loops ↔ MLP effective rank 1872 covered at only 7–13%.
- Pragmatic trade-off: for multi-step workflows with conditional branches and cross-turn state (≥14 nodes, dense branching), the core model should get full FT (3B runs on a single GPU; 8B needs 1–2× A100; train once, a weight-level "compile," inference cost two orders of magnitude below orchestrated agents). For pure instruction/style/single-turn retrieval, or compute-constrained settings, keep LoRA + an external state machine.
- Evaluation metrics: drop the misleading "conversation completion rate" proxy; track terminal-state reach rate, branch coverage, state-tracking accuracy, and step-level correctness instead.
- Five practical recommendations: ① decouple routing from generation (a small fully-FT model handles state and routing; LoRA handles style); ② externalize the state machine (write workflows as explicit graphs/DSL; the model outputs intent, not direct execution); ③ specialize small models (full FT for procedural tasks, LoRA for conversational ones); ④ regression tests for conditional branches (LLM-as-user multi-turn simulation before deployment; block on failure); ⑤ pin versions and record training topology.
- arXiv:2607.21612 | University of Melbourne | 14 pages | submitted 2026-07
- Task Success: LoRA ≤2.54 vs Full FT 4.11 (p<0.001)
- Conversation completion rate: LoRA 95.5–99.0% vs 100%
- Effective rank: Travel 888 / Zoom 761 / Insurance 1026 (MLP down_proj up to 1872)
- Energy captured at r=128: 42.5% / 50.8% / 43.0% (only 43–51%)
- Cross-domain gap: 0.8–2.2 points, largest on Insurance
2. The Intuition: What "Low-Rank" and "Procedural Knowledge" Mean
Mathematically: full fine-tuning's weight update ΔW has an average effective rank of 761–1026 (Insurance's MLP down_proj reaches 1872), while LoRA even at r=128 can capture at most 43–51% of the squared Frobenius energy, per the Eckart–Young theorem. A large gap.
3. What the Experiments Show (Key Numbers)
Three task domains: Travel booking (14 nodes), Zoom support (14 nodes), Insurance claims (55 nodes, ~4× complexity). Models: Qwen2.5-3B / Qwen3-8B. Scoring: Claude Sonnet 4.5 as primary judge + GPT-4.1 for cross-checking (dual judges).
| Metric | LoRA (r=16–128) | Full FT | Significance | |---|---|---|---| | Task Success (out of 5) | ≤ 2.54 | 4.11 | p<0.001, \|d\|>1.9 | | Conversation completion rate | 95.5–99.0% | 100% | — | | Cross-domain gap (mean of r=32/128) | 0.8–2.2 points behind | — | Largest gap on Insurance |
A glaring paradox: LoRA speaks smoothly (near-perfect completion rates) but walks the wrong path (only half the success rate). The authors call this *completion without correctness*. Stranger still: higher rank (>32) actually degrades scores (2.54→2.44→2.10); §3.4 shows LoRA has *lower* held-out loss (0.81 vs 0.89) yet worse behavior — "LoRA overfits without learning": it memorizes what conversations look like, not why they should unfold that way.
4. Three Perspectives Integrated
Perspective A — Mechanism: Why the low-rank bottleneck breaks on procedural knowledge
Perspective B — Review: The title is too bold; mechanistic extrapolation overreaches
The behavioral results are credible (systematic ablations + cross-domain replication + dual judges; rated a *clean, practical negative result*), but the scope is narrow:
Perspective C — Architecture: Moving past surface fluency
5. Final Verdict
1. Trust the behavior, doubt the title. The paper's greatest value is a clean, reproducible empirical demonstration that naive uniform LoRA systematically underperforms full FT on multi-step procedures, precisely locating the failure at the behavior layer rather than the loss layer. This finding should be adopted. 2. Don't treat it as iron law. The claim of "procedural knowledge is not low-rank / a fundamental limitation" is under-evidenced; heterogeneous-rank variants, DoRA, and others may yet flip the result. Cite it as an "empirical negative result," not a "representation impossibility." 3. A lesson for agent architectures: any core workflow that must "walk the right path" should not have its correctness resting on naive LoRA weights; either use full FT or externalize routing/state. Self-deceiving with "conversation fluency" is the biggest trap this paper exposes. 4. Evaluation must change: replace "didn't crash = success" completion rates with terminal-state reach rate, branch coverage, and cross-turn state accuracy.