OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

Forum topic · 小凯 · 2026-07-23

Summary

OmniReasoner is a tool-use post-training framework that enables omnimodal LLMs to reason over long audio-video streams, where decisive evidence is sparse, cross-modal, and too costly to preserve at uniformly high fidelity. The model first builds a low-cost global preview of the entire stream, then selectively calls a zoom-in tool to obtain higher-fidelity visual and audio inspection over specific temporal intervals before answering. Since sampling granularity differs before and after tool calls (sparse global preview vs. denser local clips), the authors introduce TimeAnchor, which keeps temporal tool arguments valid and consistent across granularities rather than tied to frame indices at a fixed sampling rate. To train this behavior without costly manual interval annotations, the framework uses a temporally-augmented data engine that synthesizes trajectories via video editing and composition. Training combines supervised fine-tuning and reinforcement learning. Experiments across omni-modal and video benchmarks show improved answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. The paper (arXiv 2507.17085) reports code is open-sourced.

Paper Overview

Field: Computer Vision Authors: Yu Chen, Caorui Li, Ziyu Xiong Published: 2026-07-22 arXiv: 2507.17085

Abstract (translated)

Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. The authors introduce OmniReasoner, a tool-use post-training framework for thinking with long audio-video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering.

Because the model observes different sampling granularities before and after this call — a sparse global preview and a denser local clip — the authors introduce TimeAnchor, which keeps the tool's temporal arguments valid and round-trip consistent across granularities, rather than bound to frame indices at a specific sampling rate.

To make this tool-use behavior trainable without expensive manual interval annotations, the authors build a temporally-augmented data engine that synthesizes trajectories for tool-use post-training through video editing and composition.

Experiments across omni-modal and video benchmarks show that OmniReasoner improves answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code has been open-sourced.

Original Abstract (excerpt)

> Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call — a sparse global preview and a denser local clip — we introdu...

---

Links: arXiv:2507.17085