Constructive Circuit Amplification: A 'Minimally Invasive Surgery' for LLMs That Updates Only 1.59% of Components to Boost Math Reasoning

In the world of large language models (LLMs), the most frustrating failures are often not "doesn't know" but "knows, yet loses focus." A model can correctly count people and do the subtraction in a grade-school word problem, only to suddenly misread "6 boys are absent" as "nothing said about boys being absent" at a critical step—and then confidently produce a wrong answer, like a smart student distracted by birds outside the exam hall.

The paper "Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates" (Prakash et al., 2025) does something that sounds like sci-fi surgery: rather than retraining the whole model, it first finds the word where reasoning first goes astray, then locates which attention heads and MLP neurons are "pushing it toward the right path," and finally updates only this tiny set of components (as little as 0.17%, at most 1.59%). This improves math reasoning accuracy by up to +11.4% while barely affecting general capabilities like MMLU, TriviaQA, and TruthfulQA.

This deep-dive focuses on DCM masking (Desiderata-based Component Masking) and targeted sparse sub-network updates: how it finds "which neurons to touch," why updating so little works, what the experiments show, and where its boundaries lie.

---

🧭 1. Why "touching only a little" may beat a "full-body massage"

The paper builds on two empirical facts from mechanistic interpretability:

First: capabilities are not uniformly distributed across parameters. Many studies show certain behaviors are dominated by sparse subnetworks—circuits—composed of a few attention heads and MLP neurons working together. Classic examples include indirect object identification, greater-than comparison, and entity tracking (Wang et al., 2022a; Hanna et al., 2023; Prakash et al., 2024).

Second: fine-tuning gains often "reinforce existing circuits" rather than inventing new mechanisms (Jain et al., 2023; Prakash et al., 2024; Chhabra et al., 2025). In other words, the model already "almost knows"—the correct internal pathway is just not strong enough, or gets outcompeted by noisy interfering pathways (Rai et al., 2025; Ortu et al., 2024).

> Tip | What is "circuit competition"? > Think of the model's internal computation as multiple parallel candidate reasoning routes: some push toward the correct answer, others introduce plausible-but-wrong shortcuts. Which token gets output is the result of these routes "tugging" at the logits level. CCA's core idea: make the correct route louder so it wins the race for the next token.

If we can identify the small set of components most critical for correct reasoning and update only their parameters, we can: 1) Pull math reasoning back on track; 2) Minimize disturbance to other capabilities (the vast majority of parameters stay frozen).

That is the strategic intuition behind CCA (Constructive Circuit Amplification).

---

🧪 2. CCA pipeline: find the wayward word, find the corrective circuit, then update only them

CCA has three steps (Figure 1):

1. Token Localization: given a pair of correct/incorrect reasoning traces, find the pivotal token where the wrong trace starts to diverge, and select the token before it as the intervention point. 2. Model Component Localization: use DCM to learn a sparse binary mask identifying which attention heads and MLP neurons most strongly "push toward the correct token and suppress the wrong token." 3. Targeted Parameter Updates: apply gradient updates only to masked components (everything else frozen), in very few steps (50).

This article focuses on steps 2 and 3: how is the DCM mask learned? How do sparse updates work? Why is this more "precise" than LoRA?

---

🧩 3. The DCM mask: turning "the next word we want" into an optimizable wish list

🧷 3.1 From reasoning traces to an Error-Localization dataset: DCM's training fuel

DCM is not trained directly on raw math problems, but on a purpose-built, highly "acupuncture-style" dataset. Each sample has three parts (Figure 3):

• prefix: the shared reasoning prefix, up to and including the intervention token
• desired_token: the next token in the correct reasoning trace
• undesired_token: the next token in the incorrect reasoning trace

In other words, DCM does not learn "what is the answer to this problem" but a finer local objective:

> Given the prefix, make the model prefer generating desired_token over undesired_token.

This is crucial: it converts a long chain-of-thought problem into a "next-token preference problem," allowing direct training of the mask and subsequent updates via logit differences.

🧠 3.2 What DCM does at the component level: not changing weights, but learning "should this double?"

DCM's goal is to identify, without changing model weights, which components' outputs favor the desired token. The paper defines a masked intervention (Equation 1):

\[h_{\text{org}} = m_i \cdot 2 \cdot h_{\text{org}} + (1-m_i)\cdot h_{\text{org}}\]

where:

• \(h_{\text{org}}\) is the original output of a component (attention head or MLP neuron);
• \(m_i\) is the mask value for that component (tending to 0/1 after training).

If \(m_i=1\), the component's output is amplified 2×; if \(m_i=0\), it stays unchanged.

Intuitively, DCM is doing an "internal volume adjustment": turn some components up, see if the next token moves toward the desired one. If amplifying a component raises the desired logit and lowers the undesired logit, it is likely part of the "constructive circuit."

> Tip | Why the crude "multiply by 2"? > This is not about fine-grained control but identifiability: a fixed-gain test makes it clearer which components are "correct-direction promoters." Permanent strengthening happens only afterward, during parameter updates.

🎯 3.3 DCM's objective: maximize the desired-vs-undesired logit gap while enforcing sparsity

The mask loss (Equation 2):

\[L = -(\text{logit}_{\text{desired}} - \text{logit}_{\text{undesired}}) + \lambda \sum m\]

Breaking it down:

• The first term maximizes \(\text{logit}_{\text{desired}} - \text{logit}_{\text{undesired}}\). This is very causal: it does not require generating the whole correct reasoning—only taking the right direction at the next step after the intervention point.
• The second term \(\lambda \sum m\) is an L1 sparsity penalty on the mask, pushing the selection toward as few components as possible.

\(\lambda\) clearly encodes: how many components are you willing to pay for a stronger logit gap. The paper sweeps \(\lambda \in \{1e\text{-}2, 5e\text{-}3, 1e\text{-}3, 1e\text{-}4\}\) to pick optimal sparsity (Table 4).

Training details (Section 3.1.2 & Appendix C):

• Adam, learning rate 5e-3
• batch size 8, 50 epochs
• early stopping: if the mask stops changing after 20% of batches within an epoch
• clamp mask to [0,1] after each gradient update (for compatibility with Equation 1)
• interventions implemented with NNsight

This makes DCM a "sparse circuit probe": across many prefix-local scenarios, it finds a stable, reusable component set that maximizes the "correct token" advantage.

📌 3.4 Which components does the mask cover? Q/K/V heads and MLP neurons

The DCM mask spans multiple component types:

• Q/K/V of attention heads (including grouped-attention key/value head counting structure)
• MLP neurons

Table 7 reports selected component counts (mean ± variance) across models and token localization methods (Prefix/Branching). A striking pattern: far more MLP neurons are selected than heads—e.g., Gemma-2-2B-It under Branching selects about 3969±398 MLP neurons vs. a few hundred Q/K/V heads. This hints at a mechanism:

• attention heads handle "information routing and reading";
• MLP neurons handle "nonlinear pattern/rule processing."

For math reasoning—converting textual conditions into constraints and pushing logic forward—the MLP side may carry much of the "rule transformation" work. The paper keeps this as an empirical report without causal dissection, but it shows: CCA's sparsity does not mean "touching only a few heads"—it may touch a small set of heads plus a small set of neurons.

---

🩺 4. Sparse updates: only after the mask is learned does the actual "surgery" begin—and only on selected spots

🔧 4.1 Update objective: still the desired/undesired logit gap

In step 3 (Section 3.1.3), parameter updates use the same local objective: make desired_token dominate undesired_token, described as the "negative logit difference between desired and undesired tokens" (consistent with DCM's desiderata).

Key difference:

• DCM stage: no parameters change, only the mask is learned ("which components would help if amplified").
• Update stage: mask fixed, gradient updates applied only to masked components' parameters (all else fully frozen), permanently reinforcing the effective components.

Training strategy:

• 50 gradient steps total
• validation every 2 steps (up to step 10), then every 10 steps
• test on the best validation checkpoint
• learning rate sweep: {1e-2, 5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5}
• because the data is small and few updates are expected: full-dataset gradients, not mini-batches (explicitly stated)

This resembles "few-step, stable, clearly directed" fine-tuning: not migrating far in parameter space, just nudging a few components toward preferring the correct token.

🧠 4.2 Why the name "Constructive Circuit Amplification"?

Every word is literal:

• Circuit: the sparse component set selected by the DCM mask.
• Amplification: both the DCM "2× amplification test" and the long-term post-update enhancement make this circuit stronger in internal competition.
• Constructive: these components promote correct reasoning, not merely correlate with output—because the objective contrasts desired vs. undesired, not "make output look like some answer."

This matters: traditional fine-tuning often reinforces "correlated but unreliable" shortcuts together. CCA tries to restrict reinforcement to the computation that pulls the model from the wrong branch back to the right one.

---

📈 5. Reading the results: how much do sparse updates help? Does the mask matter?

🧮 5.1 On GSM-Symbolic: up to +11.4%, touching 0.13%–1.59% of components

Table 1 is the core results table. Focus on "CCA w mask":

• Gemma-2-9B-Instruct: original 0.807 → Branching + mask 0.881 ±0.015 (+7.4%), updating 0.17% of components
• Gemma-2-2B-Instruct: original 0.411 → 0.525 ±0.010 (+11.4%), updating 1.59% (the paper's maximum ratio)
• OLMo-2-1124-13B-Instruct: 0.742 → 0.786 ±0.005 (+4.4%), 0.44% of components
• OLMo-2-1124-7B-Instruct: 0.739 → 0.794 ±0.006 (+5.5%), 0.25% of components

🧷 5.2 Mask vs. no mask: the mask doesn't always win, but it makes "controllability" more credible

Table 1 also ablates CCA w/o mask (skipping DCM, allowing broader components in updates). Interestingly, w/o mask is sometimes slightly higher—e.g., Gemma-2-2B Branching: 0.532 (w/o) vs. 0.525 (w). This reminds us that DCM's value is not necessarily "a higher peak" but: 1. near-comparable gains with fewer changes; 2. adherence to a minimal-intervention principle, theoretically less damaging to general capabilities; 3. moving the method from empirical fine-tuning toward "mechanism-aligned targeted enhancement."

If your deployment cares deeply about "other capabilities must not degrade," the mask matters more.

🌿 5.3 How well are general capabilities preserved? Basically "almost untouched"

Table 2 reports absolute differences vs. the original model on five benchmarks. The overall picture: side effects are generally small, mostly around ±1 points, with an isolated -4.0 on TriviaQA (Gemma-2-9B Prefix w/o mask); with the mask, such extreme drops are not prominent. This matches CCA's design logic: touching very few components limits disturbance to other tasks.

---

🧠 6. Putting DCM and sparse updates together: they solve "where to cut"

Traditional parameter-efficient fine-tuning (e.g., LoRA) solves "adapt with fewer added parameters." CCA targets a different dimension:

> Not "how many parameters to add," but "which internal mechanisms of the original model must change."

The DCM mask gives a mechanistically located answer: at the wrong-vs-right branch point, which components' amplification most pushes the desired token? This is more like a surgeon reading an imaging scan than "bolting a LoRA adapter onto every layer": you don't strengthen every muscle—just release the spot compressing the nerve.

The sparse update then turns this diagnosis into treatment: gradient updates on only those components, fixing their preference in place.

---

⚠️ 7. Limitations and realism: not a "universal repair tool," but a replicable paradigm

The paper's own Discussion admits limits (faithfully relayed, with some interpretation):

1. Validated mainly on math reasoning (GSM-Symbolic); generalization to code, scientific reasoning, or multimodal tasks is untested. 2. Building the Error-Localization dataset requires correctness signals and multiple generations: paired correct/incorrect traces are needed; real tasks may lack clear answers or have expensive annotation. 3. Only a single round of targeted enhancement: real deployments often involve multi-round fine-tuning and continual learning with catastrophic forgetting; CCA may be "safer," but how to compose multiple CCA rounds is unexplored.

These limits don't diminish the contribution—they highlight its core value: pushing "mechanism localization → sparse updates" from single-step output tasks to long chain-of-thought reasoning, with empirical proof that "updating a small fraction of components can significantly improve a specific capability" is not just a slogan.

---

🧾 References (5 core sources, cited in or constituting the paper)

1. Prakash, N., Ren, D., Moritz, D., & Assogba, Y. (2025). Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates. arXiv:2512.16914v1. 2. Davies, X., et al. (2023). Desiderata-based Component Masking (DCM). 3. Wang, K., et al. (2022a). Mechanistic interpretability / circuits in transformers. 4. Hu, E., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. 5. Mirzadeh, I., et al. (2025). GSM-Symbolic.