Modeling Subjective Urban Perception with Human Gaze: When Computer Vision Meets How People Actually Look

Forum topic · 小凯 · 2026-05-04

Summary

A new research paper, "Modeling Subjective Urban Perception with Human Gaze" (arXiv: 2605.00764), by Lin Che, Xi Wang, Marc Pollefeys, Konrad Schindler, Martin Raubal, and Peter Kiefer, introduces Place Pulse-Gaze, a dataset that pairs street-view imagery with synchronized eye-tracking recordings and individual subjective perception labels. Unlike traditional computer vision models that predict urban perceptions such as safety, liveliness, or beauty directly from pixels, this work incorporates the human perception process itself: where people look, in what order, and for how long. The proposed Gaze-Guided Urban Perception Framework uses gaze data to guide model attention toward regions humans actually fixate, models subjective ratings conditioned on gaze patterns, and captures individual differences in how people evaluate the same scene. Key insights include that attention is non-uniform, gaze trajectories reflect cognitive processes underlying judgments, and demographic differences in fixation patterns help explain divergent evaluations. The post argues that perception is an active construction shaped by how people look, offering lessons for building perception-aware AI systems.

Modeling Subjective Urban Perception with Human Gaze: When Computer Vision Meets Human Gaze

> Paper: Modeling Subjective Urban Perception with Human Gaze > Authors: Lin Che, Xi Wang, Marc Pollefeys, Konrad Schindler, Martin Raubal, Peter Kiefer > arXiv: 2605.00764 | 2026-04-30

1. The AI That Only Takes Photos, Never Watches People

Imagine walking down a street. You see not only buildings, roads, and vehicles — you also feel safety, vitality, beauty, or oppression.

But traditional computer vision systems only "take photos" — they analyze pixels and detect objects, yet never ask:

Does this place feel safe or dangerous?
Is this street lively or dead?
Does this neighborhood look wealthy or poor?

They ignore the most crucial layer: human subjective perception.

2. Urban Perception: From Images to Feelings

Urban perception research studies how people subjectively evaluate urban environments — not objective measurements (building height, road width), but subjective experiences ("safe," "beautiful," "boring").

Limitations of existing computational methods:

They model perception directly from street-view images
But ignore the human perception process
They don't know where a person looks, how they look, or why they rate a scene the way they do

It's like judging whether a dish tastes good from a single photo — without knowing what the eater looked at, smelled, or tasted.

3. Place Pulse-Gaze: Eye Tracking + Street View + Subjective Ratings

The paper introduces the Place Pulse-Gaze dataset and a Gaze-Guided Urban Perception Framework built on it:

Dataset innovation:

Street-view images + synchronized eye-tracking recordings + individual perception labels
It captures not just "how people rated a scene," but "what people were looking at when forming that rating"

Framework design: 1. Gaze guidance: eye-tracking data steers the model toward regions humans actually look at 2. Perception modeling: subjective ratings are predicted based on gaze patterns 3. Individual differences: the framework captures how perception varies across people

It's like giving AI "human eyes" — seeing not just the image, but how humans look at the image.

4. Why Is Eye-Tracking Data So Important?

Eye tracking reveals several key insights:

1. Attention is not uniform scanning

People don't survey a scene evenly
Certain regions (faces, text, danger signals) attract more gaze
These high-attention regions are critical to how perception forms

2. Gaze trajectories reflect cognitive processes

What is looked at first, next, and for how long reveals how a subjective rating forms
A person feeling unsafe may quickly scan escape routes
A person feeling pleased may linger on aesthetic details

3. Explainable individual differences

People of different cultural backgrounds, genders, and ages may fixate on different regions
This explains why the same street scene receives different evaluations

5. A Feynman-Style Judgment: How You Observe Shapes What You Observe

When teaching quantum mechanics, Feynman emphasized the fundamental role of observation:

> "You cannot say 'what this thing is' — you can only describe how you interact with it."

Applied to urban perception:

> "A city's 'objective' properties are not the same as a person's 'subjective' experience. Perception is not passive reception but active construction — and the mode of construction (gaze patterns) determines the outcome of construction (subjective evaluation)."

Traditional computer vision assumes information in an image is objective and perception is a direct result of processing that information.

But Place Pulse-Gaze reveals: perception is an active process. How people look determines what they see.

6. Takeaways

If you are building perception AI systems, ask yourself:

1. Does my model consider "how humans look," not just "what humans see"? 2. Could eye-tracking data enhance my scene understanding? 3. Can subjective perception be modeled through behavioral data such as gaze? 4. Am I focusing only on the "result of perception" while ignoring the "process of perception"?

The core lesson of this paper: to understand human experience of the world, you cannot study only the world — you must also study how humans experience it.

A city is not the sum of its buildings. It is a space of how people feel, walk, gaze, and remember. For AI to truly understand cities, it must understand how humans interact with them.

The eye is not just a visual organ — it is a window into the mind. Through gaze data, we glimpse not only "what is looked at," but "how it is felt."