Headline Take
The paper delivers a clean experimental finding: uniform LoRA underperforms full fine-tuning on multi-step procedural tasks. It overreaches when elevating this negative result to a theorem-style claim that "procedural knowledge is fundamentally not low-rank." Take the behavioral discovery seriously; treat the title-level claim with caution.
What the Paper Is
- Title: *Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures*
- Authors: Simon Dennis, Kevin Shabahang, Hao Guo, Rivaan Patil (University of Melbourne)
- Reference: arXiv:2607.21612 (v1, 14 pages, submitted July 2026)
- Core question: LoRA and other PEFT methods dominate instruction following, style transfer, and factual adaptation. Can they also match full fine-tuning for procedural knowledge — the ability to traverse conditional branches and multi-step paths to a terminal state?
- Full fine-tuning rewrites the entire "cookbook" of weights.
- LoRA adds only a small margin note (ΔW = B·A with tiny rank r) and expects it to retune the whole recipe.
- Instruction following and style transfer are local, smooth flavors — two margin lines suffice.
- Procedural knowledge is a sequential dance: pick a path, decide the next step from prior context, loop until a terminal state. This dance spans the whole recipe and is tightly coupled — two margin lines cannot capture it.
- Completion rate measures surface token distribution; success rate measures state→action mapping. The two are decoupled.
- Low-rank B·A can memorize high-frequency n-grams and templated utterances, but struggles to encode the conditional transitions of a hidden state space (effective rank 888+).
- Why high rank degrades: decision trees are sparse, local, and conditional (86–2,381 paths, nested loops). Low-rank spaces express only smooth, globally-correlated perturbations. As r grows, capacity is wasted fitting the wording of specific paths — overfitting the surface rather than the abstract transition rule.
- MLP is the real bottleneck: state→action mapping lives in MLPs, whose effective rank is highest (down_proj 1872) and whose capacity gap is largest. K/V projections appear "sufficient" only because GQA caps their maximum rank in the hundreds — a dimensionality artifact, not evidence that K/V carries procedural knowledge.
- Only uniform-rank vanilla LoRA is tested; heterogeneous-rank variants (e.g., AdaLoRA) are not.
- The title ("not low-rank / fundamental") overshoots the evidence.
- The SVD analysis is load-bearing: it treats the full-FT ΔW as the target LoRA must approximate. But Eckart–Young only bounds approximation of that specific ΔW — it does not prove the non-existence of any low-rank update that reproduces the behavior. The paper's own Limitations section acknowledges this.
- Hyperparameters: α=2r, learning rate 2e-4 for LoRA vs 2e-5 for full FT (a 10× advantage for LoRA without per-rank tuning); effective-rank threshold never defined.
- LLM judges without human verification; only Qwen family; only three customer-service domains.
- Variants not ruled out: AdaLoRA / heterogeneous rank, LoRA+, DoRA, VeRA, QLoRA, per-rank tuning — any of these closing the gap to full FT at r≤128 would refute "structurally infeasible."
- The surface-fluency trap: today most agents (LoRA backbone + external prompt/tool orchestration) push procedural correctness onto prompts and schedulers. Wherever the model must internalize conditional branches and cross-turn state, a rank-capped LoRA (≤128) captures only 43–51% of energy and will walk the wrong path. Failure modes map one-to-one: hallucinated steps ↔ overfit dialog patterns; lost state ↔ late-turn state loss rebound; infinite loops ↔ MLP effective rank 1872 covered only 7–13%.
- Pragmatic trade-off: for multi-step, branch-heavy, cross-turn-state flows (≥14 nodes, dense branches), put Full FT at the core (single-GPU for 3B; 1–2× A100 for 8B; one training run yields "compiled" weights; inference cost can be two orders of magnitude lower than orchestrated agents). For pure instruction / style / single-turn retrieval, or compute-constrained cases, keep LoRA plus an external state machine.
- Better metrics: drop "dialog completion rate" as a misleading proxy; track terminal-state reach rate, branch coverage, state-tracking accuracy, and step-level correctness.
- Five practical rules: (1) decouple routing from generation — a small fully-FT model handles state and routing, LoRA handles style; (2) externalize the state machine — write the flow as an explicit graph/DSL and have the model emit intents, not direct actions; (3) specialize small models — full FT for procedural flows, LoRA for conversational ones; (4) regression-test conditional branches before deployment (LLM-as-user multi-turn simulation, block on failure); (5) lock versions and record training topology.
- arXiv:2607.21612 | University of Melbourne | 14 pages | July 2026 submission
- Task Success: LoRA ≤2.54 vs Full FT 4.11 (p<0.001)
- Dialog completion: LoRA 95.5–99.0% vs 100%
- Effective rank: Travel 888 / Zoom 761 / Insurance 1026 (MLP down_proj up to 1872)
- r=128 energy captured: 42.5% / 50.8% / 43.0% (only 43–51%)
- Cross-domain gap: 0.8–2.2 points, largest on Insurance
Intuition: What "Low-Rank" and "Procedural" Mean
Mathematically: full-FT weight updates have mean effective rank 761–1026 (MLP down_proj in Insurance reaches 1872), while LoRA at r=128, by the Eckart–Young theorem, captures only 43–51% of squared Frobenius energy — far short of the full update.
What the Experiments Show (Key Numbers)
Tasks: Travel booking (14 nodes), Zoom support (14 nodes), Insurance claims (55 nodes, ~4× complexity). Models: Qwen2.5-3B / Qwen3-8B. Scoring: Claude Sonnet 4.5 as primary judge, GPT-4.1 cross-review (dual-LLM judge).
| Metric | LoRA (r=16–128) | Full FT | Significance | |---|---|---|---| | Task Success (out of 5) | ≤ 2.54 | 4.11 | p<0.001, |d|>1.9 | | Dialog completion | 95.5–99.0% | 100% | — | | Cross-domain gap (r=32/128 mean) | 0.8–2.2 points behind | — | Insurance shows the largest gap |
The Core Paradox
LoRA "talks the talk" (near-perfect completion) but "walks the wrong path" (success rate near half). The authors call this *Completion without correctness*.
Even stranger: higher rank hurts. Beyond r=32, scores fall (2.54 → 2.44 → 2.10). Section 3.4 shows LoRA achieves a lower held-out loss (0.81 vs 0.89) yet behaves worse — "LoRA overfits without learning": it memorizes the *look* of dialog without learning *why* it should walk that way.
Three Lenses on the Result
Lens A — Mechanism: Why Low Rank Breaks on Procedural Knowledge
Lens B — Review: Title Too Bold, Mechanism Over-Extended
The behavioral results are credible (systematic ablation + cross-domain replication + dual-LLM judge; integrity review rated it a *clean, practical negative result*), but the boundary is clear:
Lens C — Architecture: Beyond Surface Fluency
Final Verdict
1. Trust the behavior, doubt the headline. The paper's chief value is a clean, reproducible empirical demonstration that uniform LoRA is systematically inferior to full FT on multi-step procedures, and a precise diagnosis that the failure occurs at the behavioral level rather than the loss level. Adopt this finding. 2. Do not treat it as a law. The "procedural knowledge is not low-rank / fundamentally constrained" claim is under-supported; heterogeneous-rank variants and methods like DoRA may still close the gap. Cite this work as an *empirical negative result*, not as evidence of representational impossibility. 3. Architectural warning: any agent whose core workflow must "walk the right path" cannot place correctness on the shoulders of vanilla LoRA weights. Either full FT the core, or externalize routing/state. Self-deception via dialog fluency is the largest trap this paper exposes. 4. Reform evaluation: replace completion-rate vanity metrics with terminal-state reach rate, branch coverage, and cross-turn state accuracy.