MACRO: Markov-Chain Layer Routing for Transformers

A Transformer can sometimes encode the correct answer in an intermediate layer, only to lose or suppress that signal as processing continues toward the output. The MACRO method, presented by Batorskq and colleagues in August 2026, explores whether inference can be improved by changing the order in which existing layers are executed—without changing model weights.

Paper: https://arxiv.org/abs/2608.05872 Code: https://github.com/Batorskq/MACRO HTML version: https://arxiv.org/html/2608.05872v1

The problem: correct signals can disappear

In a standard Transformer, layers run sequentially from layer 1 to layer \(N\). Each layer receives the output of its immediate predecessor. Yet previous observations show that a model's probability for a correct answer can be higher at an intermediate layer than at the final layer. This suggests that useful information exists during computation but may be weakened or overwritten later.

The earlier Dr. LLM approach addressed this problem by searching for routes that could skip, repeat, or normally execute layers. Although effective, its search space was large, requiring 14.8 hours to identify a route.

MACRO’s core idea

MACRO models layer transitions as a Markov chain. Each decision about which layer to execute next depends on the current state rather than the complete execution history. The resulting transition probabilities are optimized, and the Viterbi algorithm is used to find a high-scoring route through the layered model.

Instead of enumerating every possible skip, repeat, and normal-execution combination, MACRO structures the search as a dynamic-programming problem. The paper reports that this reduces route-search time from 14.8 hours to 1.6 hours—a 9.4× speedup.

Main results

The experiments cover five models:

• Llama3-8B
• Qwen3-1.7B
• Qwen3-4B
• Qwen3-7B
• Mistral-7B

The models were evaluated on seven benchmarks:

• GSM8K
• MATH500
• OpenBookQA
• ARC-C
• ARC-E
• CommonsenseQA
• HellaSwag

Reported aggregate results include:

• An average improvement of 5.0% over the non-routing baseline
• An average improvement of 7.2% over Dr. LLM
• A 9.4× route-search speedup

The most notable individual result is Qwen3-1.7B on GSM8K, where performance rises from 43.4% to 69.5% by rewinding layer 7 to layer 3. This gain comes from changing layer execution rather than fine-tuning, adding data, or altering the architecture.

Route transfer

Routes discovered on GSM8K, a mathematical word-problem benchmark, also improve results on MATH500, a mathematical competition benchmark. Routes found on CommonsenseQA, however, do not transfer effectively to mathematical reasoning. This pattern indicates that routes may capture structures associated with a task family rather than arbitrary routing noise.

The authors also report that a shared route trained on multiple tasks can transfer to tasks not used during route training. For Qwen3-1.7B, this approach produces an average out-of-domain improvement of 9.02%. The result suggests that a broadly useful route may sometimes replace task-specific route searches, although its transferability is not universal.

Why routing can help

MACRO’s interpretability analysis argues that the standard forward pass can contain “suppressing” layers that weaken correct-answer signals and “enhancing” layers that preserve or strengthen them. A route can avoid the former or revisit the latter, keeping useful intermediate information available at the output.

The benefits appear to depend on model size. The paper reports larger gains for smaller models, while Llama3-8B improves less than Qwen3-1.7B. One possible interpretation is that larger models may already internalize some routing behavior in their weights, whereas smaller models benefit more from making that behavior explicit at inference time.

This result connects with earlier work on information flow in models such as Othello-GPT and the Toy Model of Superposition, where intermediate representations encode substantial information that is not always fully reflected in the final output.

Limitations

• Search is still slow: A 1.6-hour search is much faster than Dr. LLM’s reported 14.8 hours, but it is unsuitable for many real-time applications. Once found, the route is used like an ordinary forward pass.
• Transfer has boundaries: Mathematical routes transfer across mathematical tasks, but routes do not reliably transfer from mathematics to commonsense reasoning.
• Benefits may diminish with scale: Smaller models show larger gains, which may limit the method’s relative value for already-capable large models.
• The Markov assumption is approximate: Assuming that the next transition depends only on the current layer can miss non-Markov routing patterns. Viterbi search is therefore an approximation, even though the reported experiments find it effective.

Broader significance

MACRO challenges the assumption that layers must always execute in their original order from 1 to \(N\). Most model improvements modify learned parameters through pretraining, fine-tuning, reinforcement learning, or preference optimization. MACRO instead changes the inference-time execution path while keeping the weights fixed.

This reframes inference as a scheduling problem. A model may contain useful intermediate computations, but the standard sequential route may not be the best way to expose them. By allowing layers to be skipped, repeated, or reordered according to a learned transition policy, MACRO offers a way to unlock capabilities already present in the model.

Its central insight is simple: the model may not lack the answer; the fixed forward pass may obscure it. Although MACRO does not establish a universally optimal Transformer execution order, it provides evidence that layer routing can be a practical complement to weight-based optimization.