Keep Yelling Assistant: Vision-Language Pipeline for Emotional Reactions to Risky Driving

Forum topic · 小凯 · 2026-07-21

Summary

Researchers Harine Choi, Eun Hak Lee, and Zhengzhong Tu introduce Keep Yelling Assistant (KYA), a vision-language pipeline that detects risky driving behaviors in real time and generates emotionally expressive verbal responses tailored to driver preferences. The vision module uses YOLOv8 variants to detect nearby vehicles and identify dangerous maneuvers such as sudden cut-ins, extracting and normalizing metrics like relative distance, speed, and projected reach time into a structured behavior log. The language module processes this log with user-defined emotional tone settings (neutral, humorous, analytical) and generates responses via state-of-the-art LLMs including ChatGPT-4o, Claude 3, Gemini 2.5, and Copilot. The system was evaluated on dashcam videos and a user study with 108 participants. All models received favorable ratings, with the YOLOv8s + ChatGPT-4o combination achieving the highest score of 4.29 out of 5.00. KYA proposes a new paradigm for emotionally intelligent in-vehicle AI. Paper: arXiv:2507.15486.

Paper Overview

Research Area: cs.CV Authors: Harine Choi, Eun Hak Lee, Zhengzhong Tu arXiv: 2507.15486

Introduction

This study introduces a vision-language pipeline that detects risky driving behaviors and generates emotionally expressive responses to support driver awareness and comfort. Although vision-language models have advanced perception and reasoning in autonomous driving, existing systems rarely consider the emotional dimension or real-world user experience.

System Design

Keep Yelling Assistant (KYA) detects high-risk driving maneuvers in real time, such as sudden cut-ins, and produces emotional responses through a large language model tailored to driver preferences. The framework comprises two core modules:

Vision module: Uses YOLOv8 variants to detect nearby vehicles and identify risky behaviors such as sudden cut-ins. Key driving metrics — relative distance, speed, and projected reach time — are extracted and normalized to produce a structured behavior log.
Language module: Processes the behavior log with user-defined emotional tone settings (e.g., neutral, humorous, analytical) and generates verbal reactions using state-of-the-art LLMs including ChatGPT-4o, Claude 3, Gemini 2.5, and Copilot.

Evaluation

The system was evaluated using dashcam videos containing risky driving behaviors and a user study involving 108 participants. Participants selected preferred response styles, and the LLMs were evaluated based on emotional alignment.

All models received favorable ratings, although preferences varied across personas.
The combination of YOLOv8s + ChatGPT-4o achieved the highest score: 4.29 / 5.00.

Conclusion

By integrating real-world perception with emotionally adaptive dialogue, KYA introduces a new paradigm for emotionally intelligent in-vehicle artificial intelligence.

Paper link: https://arxiv.org/abs/2507.15486