Policy-based Foveated Imaging and Perception: Task-Aware Imaging at Acquisition Time

Forum topic · 小凯 · 2026-06-03

Summary

This paper (arXiv:2506.00009) by Howard Xiao, Jan Ackermann, and Boyang Deng introduces a real-time, predictive, task-aware foveated imaging system that operates directly at image acquisition time. Ultra-high-resolution sensors can capture fine spatial details needed by visual perception tasks, but acquiring all pixels at full resolution is often infeasible under bandwidth, latency, and power constraints. Existing strategies like spatial or temporal downsampling irrevocably discard information before task relevance is assessed. The proposed method leverages emerging dual-stream sensor architectures to dynamically allocate limited pixel bandwidth to task-relevant regions of interest while preserving low-resolution global context. The authors formalize foveated acquisition as sensor attention policy learning, where past observations guide actions determining future measurements, closing the perception-acquisition loop. Extensive simulations across multiple perception tasks show high task performance under strict pixel budgets and significant gains over bandwidth-matched baselines. The system is further validated on a 200-megapixel dual-stream sensor capturing real-world video under real bandwidth and latency constraints, demonstrating the practical feasibility of task-driven, at-acquisition foveated imaging.

Policy-based Foveated Imaging and Perception

Research area: Computer Vision Authors: Howard Xiao, Jan Ackermann, Boyang Deng Published: 2026-06-03 arXiv: 2506.00009

Overview

Ultra-high-resolution image sensors offer the potential to capture fine spatial details critical for many visual perception tasks, but acquiring and processing all pixels at full resolution is often infeasible under realistic bandwidth, latency, and power constraints. Existing approaches address this challenge through acquisition strategies such as spatial or temporal downsampling, which irrevocably discard information before task relevance can be assessed.

In this work, the authors introduce a real-time, predictive, and task-aware foveated imaging system that operates directly at image acquisition time. Leveraging emerging dual-stream sensor architectures, the method dynamically allocates limited pixel bandwidth to task-relevant regions of interest while maintaining a low-resolution global context.

Key contributions

Foveated acquisition is formalized as a sensor attention policy learning problem, in which past observations guide actions that determine future measurements, closing the perception–acquisition loop.
The system allocates pixel bandwidth to task-relevant regions while preserving global context via the dual-stream architecture.

Results

In extensive simulations across multiple perception tasks, the method achieves high task performance under strict pixel budgets and significantly outperforms related baselines operating at the same bandwidth.
The system was validated on a 200-megapixel dual-stream sensor, capturing real-world video under realistic bandwidth and latency constraints, demonstrating the practical feasibility of task-driven, at-acquisition foveated imaging.

Original abstract (excerpt)

> Ultra-high-resolution image sensors offer the potential to capture fine spatial details critical for many visual perception tasks, but acquiring and processing all pixels at full resolution is often infeasible under realistic bandwidth, latency, and power constraints. Existing approaches address this challenge through acquisition strategies such as spatial or temporal downsampling, which irrevocably discard information before task relevance can be assessed. In this work, we introduce a real-time, predictive, and task-aware foveated imaging system that operates directly at image acquisition time...

Full paper: https://arxiv.org/abs/2506.00009