Paper Overview
Field: NLP Authors: Bobo Li, Jiajia Song, Hao Fei, Wynne Hsu, Mong Li Lee Published: 2026-10-30 arXiv: 2610.42148
Summary
Decision models such as Jev have emerged as a highly promising paradigm for LLM agent systems, providing lightweight and reliable categorical reasoning for rapid System One execution. However, existing generation-oriented inference engines fail to support these workloads effectively, suffering from severe prefix recomputation overheads or distorted output probabilities.
To address this challenge, the authors propose Flide, an efficient and exact inference engine that decouples the logical tree structure of a decision request from its physical execution within a single flattened token stream.
Key Ideas
- ForkAttention: navigates request dependencies using GPU-resident indices across both standard attention and recurrent Gated DeltaNet layers, guaranteeing in exact arithmetic that every query branch's computed results are identical to independent forward passes.
- Branch topology as GPU-resident metadata: Flide adapts a compact pre-recorded execution graph to diverse request mixes.
- Checkpoint resumption: supports growing agent histories resuming from checkpoints saved in previous passes.
Results
Extensive evaluations on public hybrid models and decision models show that Flide answers queries up to 9.5x faster than vLLM and SGLang, while remaining closest to the float32 reference implementation in total variation distance. By unifying physical execution throughput with mathematical exactness, Flide lays a principled foundation for scalable, latency-sensitive decision intelligence in agent systems.
---
*Auto-collected on 2026-11-03*