From Backward Spreading to Forward Replay: A New Take on Target Construction in LLM Parameter Editing

> Paper: From Backward Spreading to Forward Replay: Revisiting Target Construction in LLM Parameter Editing > Authors: Wei Liu, Hongkai Liu, Zhiying Deng, Yee Whye Teh, Wee Sun Lee > arXiv: 2605.00358 | 2026-04-29

1. The Dilemma: Parameter Editing Is Too Complicated

Imagine you have trained a large language model and discover a problem:

• The model has memorized an incorrect fact
• "Paris is the capital of France" → correct
• "Paris is the capital of Germany" → wrong; it needs fixing

Traditional parameter editing methods:

• Compute the ideal hidden state at a target layer
• Back-propagate to earlier layers
• Edit layers jointly — this is called backward spreading

The problem: these methods are widely used, but their theoretical basis has never been systematically studied. Capability boundaries are unclear, failure modes are unknown — they are used like "black magic."

2. Systematic Analysis + An Elegant Alternative

The paper does two things:

1. Systematically studies backward spreading:

• Clarifies its capability boundaries
• Identifies practical considerations
• Uncovers potential failure modes

2. Proposes "Forward Replay":

• Simpler and more elegant
• Comparable or better performance
• Clearer theoretical grounding

Core idea of Forward Replay:

> Instead of back-propagating from the target layer, construct the target representation by "replaying" it forward through the layers.

Technical highlights: 1. Forward construction — build target representations layer by layer from the input side; more natural 2. Simplified pipeline — no complex backward computation; less error accumulation; more stable 3. Clear theory — every step is explainable, predictable, not "black magic" 4. Comparable results — experimentally validated; never worse than backward spreading, sometimes better

An analogy:

• Backward spreading = reasoning backward from the endpoint to the start
• Forward replay = walking forward from the start to the endpoint
• The latter is more intuitive and works just as well

3. Why Can "Forward" Replace "Backward"?

Problems with backward spreading:

• Error accumulation: computing backward from deep layers introduces per-layer errors that compound and grow
• Opaque theory: why does it work? What are the boundary conditions? Unclear

Advantages of forward replay:

• Intuitive and natural: aligns with the forward nature of neural network computation, flowing from input to output
• Controllable error: forward computation is more stable and does not accumulate errors
• Clear theory: each step is interpretable — you know why it works and when it might fail

4. A Feynman-Style Judgment: Simple Methods Are Often Deeper

Feynman said:

> "Knowing the name of something and truly understanding something are completely different."

In parameter editing:

> "Backward spreading has been used for years, but no one really asked 'why must it be backward?' The courage of this paper lies in questioning that 'common sense' and discovering that 'forward' works just as well — simpler and more elegant. True understanding is not following convention, but daring to ask 'why.'"

This reflects the spirit of science:

• Questioning > following
• Simplicity > complexity
• Understanding > usage

5. Takeaways

If you edit LLM parameters or research knowledge updating, ask yourself:

1. Does my editing method have a clear theoretical basis? 2. Is there a simpler, more elegant alternative? 3. Is back-propagation really the only choice? 4. Can forward construction reduce error accumulation?

The core lesson: in AI research, the most widely used method is not necessarily the best. When researchers dare to question "standard practice," systematically reveal its boundaries, and propose simpler alternatives, they show true scientific spirit. The best method is not the most complex one, but the most understood one.

In the ocean of knowledge, the forward direction is often clearer than the backward path.