Listen, Think, and Understand: Building Perception-Enabled Large Language Models with the OpenAQA Dataset

Forum topic · 小凯 · 2026-07-05

Summary

This arXiv paper (2305.10790, May 2023) by Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James Glass (MIT) addresses a key limitation of large language models: their inability to perceive and reason about sounds. The authors propose a training method that enables an LLM to perform audio-based question answering and reasoning, and they release the OpenAQA dataset of 5.6 million (audio, text) pairs to support this goal. The resulting model can describe sounds, answer open-ended questions about audio content, and reason over acoustic scenes while retaining the base model's general language capabilities. The work targets scenarios such as environmental sound understanding, audio captioning, and listening comprehension, positioning audio perception as an extension of multimodal LLMs beyond text and vision. Key contributions include the OpenAQA dataset, a lightweight audio encoder coupled to a frozen or adapted LLM, and evaluations showing strong performance on audio question answering benchmarks. This summary is based on the paper's metadata and public information; readers should consult the original PDF for exact experimental figures and implementation details.

Listen, Think, and Understand: Building Perception-Enabled Large Language Models (OpenAQA Dataset)

Source: https://arxiv.org/abs/2305.10790 (arXiv, May 2023) Authors: Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, James Glass Categories: Multimodal, Question Answering

Key points

Large language models excel at language tasks but are largely blind (and deaf) to the physical world; this paper focuses on giving LLMs the ability to listen — perceiving and reasoning about audio.
The authors propose a training method that converts LLMs into audio-understanding models capable of both closed-form and open-ended audio question answering and reasoning.
They release the OpenAQA dataset, containing 5.6 million (audio, text) pairs, as the training and evaluation resource for this capability.
The resulting model can explain sounds, answer free-form questions about audio events and scenes, and perform reasoning over what it hears — while largely preserving the underlying LLM's original language abilities.
The work targets applications such as listening comprehension, environmental sound understanding, and audio captioning, positioning audio as a next frontier for multimodal LLMs after text and vision.

Overview

While multimodal LLMs have made rapid progress in vision, the audio modality has lagged behind. This paper (commonly known as the LTU, "Listen, Think, and Understand" work) argues that a general-purpose audio-understanding LLM should go beyond classification-style tasks (e.g., AudioSet tagging or keyword spotting) and instead handle open-ended, reasoning-heavy queries: "What sounds do you hear? How do they make you feel? What might be happening?"

The approach couples an audio encoding front-end with a large language model, training on the large-scale OpenAQA corpus so the model learns to translate acoustic inputs into rich natural-language descriptions and answers. The authors demonstrate that this does not significantly degrade the base LLM's general instruction-following performance.

Evaluation

The paper reports results on audio question answering and audio captioning benchmarks, comparing against prior audio-language baselines. Note: the exact quantitative tables should be verified against the original PDF; the figures below reflect the paper's high-level claims rather than reproduced numbers.

Tasks: closed-form QA, open-ended QA, audio captioning, listening comprehension.
Training data: OpenAQA, 5.6M audio–text pairs assembled from public audio corpora and synthesized QA annotations.
Findings: the audio-enabled LLM substantially outperforms prior baselines on open-ended audio understanding while retaining language skills.

Why it matters

Extends the multimodal LLM paradigm to sound, an underexplored modality.
The OpenAQA dataset is a large public resource for audio-language research.
Demonstrates that perception can be added to a frozen or lightly-tuned LLM without destroying its general capabilities — a recipe later echoed in audio-LLM systems such as Qwen-Audio and SALMONN.

Limitations

Performance depends on coverage and quality of training audio–text pairs; rare or out-of-domain sounds may be poorly handled.
As with any LLM, hallucination risk persists when the model describes audio content.
Compute cost of the LLM backbone may constrain real-time deployment.

References

Original paper: "Listen, Think, and Understand" — arXiv:2305.10790
Related: Clotho-AQA (crowdsourced audio QA dataset), Clotho audio captioning corpus.