PoLar: Compiler-Style Optimization for LLMs — Turning Fixed Layer Sequences into Input-Specific Execution Programs

Forum topic · 小凯 · 2026-06-17

Summary

PoLar (Program-of-Layers), from the paper "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs" by Ziyue Li, Yang Li, and Tianyi Zhou (arXiv:2606.06574), rethinks LLM inference by replacing the fixed layer-0-to-D forward pass with an input-specific execution program. Instead of retraining weights, PoLar reuses frozen pretrained layers but lets each input skip redundant layers or loop specific layer segments for iterative refinement. MCTS experiments show better-than-default execution programs exist for nearly all inputs: 75.5% of correct answers have shorter paths, and 36.2% of wrong ones can be fixed with fewer layers. A lightweight 2.1M-parameter predictor (~0.01-0.06% of the base model) generates the program upfront, adding under 3.05ms overhead—less than a quarter of one LLM layer. On mathematical reasoning benchmarks, PoLar lifts Pass@5 accuracy by up to +22 points across LLaMA-3.2-3B, Qwen2.5-3B, and Qwen1.5-MoE models, with the largest gains on MoE and simpler tasks, while scaling test-time compute more effectively than plain sampling.

PoLar (Program-of-Layers) is introduced in the paper *Skip a Layer or Loop It? Learning Program-of-Layers in LLMs* by Ziyue Li, Yang Li, and Tianyi Zhou (arXiv:2606.06574). The core observation: standard LLM inference is a rigid forward pass — every input, from 1+1=? to a hard proof, traverses the same D layers in the same order. PoLar instead treats inference as a program: a per-input sequence that can skip layers or loop layer segments, with all pretrained weights frozen.

Key Ideas

Execution program as reordering, not retraining: A standard pass is π = (0, 1, ..., D-1). A PoLar program might be π = (0, 1, 2, 5, 5, 6, 7, 10, 11, 11, 12) — skipping layers 8–9, looping layer 5 and layer 11 twice. Layer functions stay fixed; only call order and count change.
MCTS validation: Monte Carlo Tree Search shows better-than-default paths exist almost everywhere:
75.5% of inputs that were correct under the default pass have shorter correct paths.
36.2% of inputs that were wrong can be corrected with fewer layers.
Allowing more loop steps monotonically raises the probability of finding a valid program (~30% at 0 extra steps → ~65% at 8 steps).
Structural bias: 54.5% of programs are single-layer segments; over two-thirds of segments are ≤2 consecutive layers; non-contiguous skips are rare (<3.2%).
Skip and loop are complementary: On LLaMA-3.2-3B (DM-1), skip-only raises accuracy from 37.9% to 45.7%, loop-only to 54.9%, but skip+loop reaches 84.7% (87.4% on Qwen2.5-3B). Looping matters more (it enables iterative refinement), but only the joint space is optimal.

Lightweight Predictor

MCTS is too slow for deployment, so PoLar trains a 2.1M-parameter network (0.01–0.06% of the base LLM): frozen embeddings → cross-attention over learnable per-layer queries → a small transformer encoder → a segmentation head (binary boundary mask grouping layers into segments) and an operation head (skip / keep / loop per segment). At inference, segmentation plus beam search over segment-level operations yields the final program. Total overhead: 3.05ms (~0.23 of one LLM layer's 13.23ms), including the predictor (0.99ms), encoder (1.95ms), and beam search (0.11ms). Crucially, the program is generated upfront — no per-layer routing during inference.

Results (Pass@5, LLaMA-3.2-3B-Instruct)

| Method | DM-1 | DM-2 | DM-3 | DM-4 | DM-5 | |---|---|---|---|---|---| | Baseline (sampling p@5) | 47.6 | 43.2 | 32.8 | 32.8 | 35.6 | | PoLar (p@5) | 68.4 | 48.0 | 46.0 | 40.4 | 45.8 |

Largest gain on the simplest split (DM-1: +20.8), indicating abundant redundant layers for easy inputs.
Cross-model: Qwen1.5-MoE-A2.7B 40.0 → 62.0 (+22.0); Qwen2.5-3B 42.2 → 59.8 (+17.6); Qwen3-8B 48.4 → 55.6 (+7.2). MoE gains most, likely because token-level expert sparsity composes naturally with layer-level path sparsity.
Test-time scaling: baseline sampling saturates (40.6 → 47.6 from k=1 to k=5), while PoLar keeps improving (46.2 → 68.4) while often using fewer unique layers — smarter compute, not more compute.

Comparison with Prior Work

| Dimension | ShortGPT / MindSkip | DR.LLM | PoLar | |---|---|---|---| | Operations | skip only | skip + loop | skip + loop | | Granularity | per-layer | per-layer | segment-level | | Decision timing | static pruning / sequential | sequential | upfront program | | Looping | none | single-layer only | multi-layer segment loops | | Optimization | none | local | beam-search global |

Open Questions

1. Validation is on mathematical reasoning; generalization to code, multimodal, and long-form generation is untested. 2. Training the predictor requires costly MCTS-derived programs. 3. The contiguity constraint on segments excludes some theoretically possible skip combinations. 4. Skipped layers break contiguous KV-cache indexing; inference engines need adaptation. 5. Repeated looping may amplify errors or diverge.

Takeaway

PoLar reframes pretrained LLMs as programmable machines: frozen weights are the instruction set, the predictor is the compiler, and each input receives a tailored execution program — fewer layers for easy inputs, more loops for hard ones, with accuracy gains up to +22 points at under a quarter-of-a-layer overhead. It is an efficiency revolution that changes *how* layers are called, not the model itself.

Reference: Li, Z., Li, Y., & Zhou, T. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. arXiv preprint. https://arxiv.org/abs/2606.06574