Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
- Paper: Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering (arXiv:2204.09634, April 2022)
- Authors: Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, Tuomas Virtanen (Tampere University)
- Category: Multimodal learning, Question Answering
- Audio clips are drawn from the existing Clotho environmental-sound corpus.
- Each clip is annotated by multiple crowdworkers with questions and answers, providing diverse phrasings and perspectives.
- Answers come in two forms:
- Yes/no (binary) answers, enabling a classification formulation;
- Open-ended, free-form answers, enabling generative/text-generation formulations.
- The dataset is publicly released together with baseline code to support reproducibility.
- Provides one of the first sizable public benchmarks for AQA over environmental sounds.
- Supports research combining audio representation learning with natural-language understanding.
- Serves as a precursor to later audio-language datasets and benchmarks (e.g., OpenAQA, audio-caption LLMs).
- Listen, Think, and Understand / OpenAQA dataset
- EA-VTR: Event-Aware Video-Text Retrieval
- ColPali: Efficient Document Retrieval with Vision Language Models
- Cross-Modal Retrieval: A Systematic Review
- Note: this forum entry's original body is a template index card; the substantive facts above are drawn from the paper's public metadata and abstract at arXiv:2204.09634. For exact dataset statistics and numerical baseline results, consult the PDF.
What the Paper Proposes
Clotho-AQA is a crowdsourced dataset for audio question answering (AQA): given an audio clip and a natural-language question about its content, a model must produce an answer. The dataset addresses the lack of large-scale public AQA corpora, which had limited progress compared to visual question answering (VQA).
Key properties:
Baseline Experiments
The authors train baseline models for both answer types:
1. Binary answers: an audio encoder (CNN-based feature extraction) combined with question text encoding, feeding a classifier for yes/no prediction. 2. Free-form answers: an audio encoder whose output is fed to a text decoder that generates the answer string.
These baselines establish reference performance levels and highlight the difficulty of grounding question semantics in non-speech audio content.