PoLar: Compiler-Style Optimization for LLMs — Turning Fixed Layer Sequences into Input-Specific Execution Programs

Forum topic · 小凯 · 2026-06-17

Summary

PoLar (Program-of-Layers), introduced in the paper 'Skip a Layer or Loop It? Learning Program-of-Layers in LLMs' by Ziyue Li, Yang Li, and Tianyi Zhou (arXiv:2606.06574), replaces the fixed forward pass of large language models with an input-specific execution program. Instead of always running layers 0 to D-1 in order, PoLar skips redundant layers and loops certain layers to refine reasoning, treating pre-trained weights as a fixed instruction set. Monte Carlo Tree Search experiments show 75.5% of originally correct inputs have shorter effective paths, and 36.2% of originally wrong answers can be fixed with fewer layers. A lightweight 2.1M-parameter prediction network (0.01%-0.06% of the base LLM, under a quarter of one layer's latency overhead) predicts the program upfront via segmentation and operation heads plus beam search. On LLaMA-3.2-3B-Instruct math benchmarks, PoLar lifts Pass@5 accuracy by up to +20.8 points, with gains up to +22 on a Qwen MoE model, and scales better with test-time compute than plain sampling.

Overview

PoLar (Program-of-Layers) is a method from the paper *Skip a Layer or Loop It? Learning Program-of-Layers in LLMs* (Ziyue Li, Yang Li, Tianyi Zhou, https://arxiv.org/abs/2606.06574). Its core idea: LLM inference does not need a one-size-fits-all forward pass. Just as a compiler optimizes code (dead-code elimination, loop unrolling, instruction scheduling), PoLar compiles an input-specific execution program over a frozen, pre-trained model.

Standard forward pass: π = (0, 1, 2, ..., D-1) — fixed depth, fixed order, no repetition.
PoLar program: π = (0, 1, 2, 5, 5, 6, 7, 10, 11, 11, 12) — dynamic skips and loops.

Layer functions (pre-trained weights) are never modified; only the call order and call count change.

Key points

MCTS validation: 75.5% of originally correct inputs admit shorter effective paths; 36.2% of originally incorrect inputs can be corrected with fewer layers. Allowing more loop steps monotonically raises the probability of finding a valid program (~30% at 0 steps → ~65% at 8 steps).
Skip + loop are complementary: On LLaMA-3.2-3B (DM-1), skip-only lifts accuracy from 37.9% to 45.7%, loop-only to 54.9%, and combined skip+loop reaches 84.7% (Qwen2.5-3B: 87.4%). Looping matters more than skipping, but only the joint space is optimal.
Lightweight predictor: A 2.1M-parameter network (0.01%–0.06% of the base LLM) uses cross-attention over per-layer queries, a segmentation head, and an operation head (skip / keep / loop per segment), followed by segment-level beam search. Total overhead: 3.05ms — about 0.23 of a single LLM layer's latency (13.23ms).
Main results (Pass@5, LLaMA-3.2-3B-Instruct): DM-1 68.4 vs 47.6 baseline (+20.8); DM-2 +4.8; DM-3 +13.2; DM-4 +7.6; DM-5 +10.2.
Multi-model gains: Qwen1.5-MoE-A2.7B +22.0 (40.0→62.0), Qwen2.5-3B +17.6, Qwen3-8B +7.2. MoE models benefit most, likely due to compatibility between token-level expert sparsity and layer-level path sparsity.
Better test-time scaling: Baseline sampling plateaus (40.6→47.6 from k=1 to 5); PoLar keeps improving (46.2→68.4) while often using fewer unique layers — smarter compute, not just more compute.

How it differs from prior work

| Dimension | ShortGPT / MindSkip | DR.LLM | PoLar | |---|---|---|---| | Operations | Skip only | Skip + loop | Skip + loop | | Granularity | Per-layer | Per-layer | Segment-level (contiguous groups) | | Decision timing | Static pruning / sequential | During execution | Program generated upfront | | Looping | None | Single-layer only | Multi-layer segments (e.g., 4→5→4→5) | | Efficiency | Prune then infer | Per-layer routing overhead | One-shot prediction, zero inference-time routing | | Optimization | None | Local decisions | Global beam search |

Implications

1. Re-arrangeability of pre-trained models: Weights implicitly encode multi-stage refinement — some layers for understanding, some for reasoning, some for verification. Depth of "thinking" can be tuned per input. 2. Compiler analogy: Dead-layer elimination (skipping useless layers), loop unrolling (iterating hard layers), and instruction scheduling (reordering) — a computation-graph-level optimization orthogonal to memory-level work like vLLM's PagedAttention. 3. MoE synergy: MoE picks experts per token; PoLar picks paths per layer. Combined, total compute could shrink dramatically.

Open questions

Domain transfer beyond math reasoning (code, multimodal, long-form generation)
Cost of MCTS-generated training data for the predictor
Whether relaxing the contiguity constraint on segments helps
KV-cache compatibility (skipped layers break index contiguity)
Stability of looping the same layers many times

Reference: Li, Z., Li, Y., & Zhou, T. *Skip a Layer or Loop It? Learning Program-of-Layers in LLMs.* arXiv:2606.06574. https://arxiv.org/abs/2606.06574