Personal Visual Context Learning in Large Multimodal Models: Personal-VCL-Bench and the Agentic Context Bank

Forum topic · 小凯 · 2026-05-13

Summary

Researchers from Google (Zihui Xue, Ami Baid, Sangho Kim) introduce Personal Visual Context Learning (Personal VCL), the prompt-time capability of large multimodal models (LMMs) to reason over user-specific visual context gathered from wearable devices such as smart glasses. To evaluate this capability, they present Personal-VCL-Bench, a benchmark covering the personal visual world across persons, objects, and behaviors. Analysis of frontier LMMs reveals a profound context utilization gap: models remain weak at leveraging visual evidence and at aggregating multiple visual observations. Motivated by these findings, the authors propose the Agentic Context Bank, a strong inference-time baseline that structures the user's visual context into a self-refining memory bank with query-adaptive evidence selection. This approach consistently outperforms standard context prompting across tasks and evaluation backbones, offering a practical path toward truly personalized LMMs. Paper: arXiv:2505.07244.

Paper Overview

Field: Computer Vision (CV) Authors: Zihui Xue, Ami Baid, Sangho Kim Published: 2025-05-09 arXiv: 2505.07244

Abstract

As wearable devices like smart glasses integrate Large Multimodal Models (LMMs) into the continuous first-person visual streams of individual users, the evolution of these models into true personal assistants hinges on visual personalization: the ability to reason over visual information unique to the wearer. The authors formalize this capability as Personal Visual Context Learning (Personal VCL) — the prompt-time capability of using user-specific visual context to resolve personalized queries.

To systematically evaluate this, they present Personal-VCL-Bench, a comprehensive benchmark capturing the personal visual world across persons, objects, and behaviors.

Key Findings

Analysis of frontier LMMs identifies a profound context utilization gap, revealing that mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain severely underdeveloped.

Proposed Method: Agentic Context Bank

Inspired by these findings, the authors propose the Agentic Context Bank, a strong inference-time baseline that:

Structures the user's visual context into a self-refining memory bank
Adopts query-adaptive evidence selection

The baseline consistently outperforms standard in-context prompting schemes across tasks and evaluation backbones, demonstrating a practical path toward future personalized LMMs.

---

*Auto-collected on 2026-05-13*