Listen, Think, and Understand: LTU and the OpenAQA Dataset (May 2023, arXiv)

Overview

This forum entry catalogs the paper Listen, Think, and Understand (arXiv:2305.10790, May 2023).

• Authors / Affiliations: Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, James Glass (MIT CSAIL / MIT-IBM Watson AI Lab)
• Source: https://arxiv.org/abs/2305.10790
• Category: Multi-modal, Question Answering, Audio-Language Models

Key Points

• The paper introduces LTU (Listen, Think, Understand), a model designed to bridge low-level audio perception (listening) with higher-level reasoning and open-ended question answering (thinking/understanding) over audio.
• It contributes the OpenAQA dataset, a large-scale collection of open-ended audio question-answering pairs built to train models on both perception and reasoning tasks.
• Motivation: conventional audio models handle recognition tasks (speech transcription, sound event classification) but lack the ability to reason about or discuss audio content in natural language the way vision-language models do for images.
• The work aligns with the broader trend of connecting pre-trained audio encoders to large language models, similar to how vision encoders are coupled with LLMs for multimodal chat.

Related Entries

• Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
• Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
• Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models

Notes

Detailed quantitative results should be verified against the original PDF at arXiv:2305.10790. Readers following the retrieval-to-generation pipeline may find this paper relevant to audio-grounded RAG and multimodal QA systems.

References

• Original paper: Listen, Think, and Understand. arXiv:2305.10790.