Finetuning Strategies for Querying Sounds by Vocal Imitation — Winning AES AIMLA 2025 Submission

Forum topic · 小凯 · 2026-08-21

Summary

This technical report by Aditya Bhattacharjee, Christos Plachouras, Sungkyun Chang, and Emmanouil Benetos (arXiv:2608.19174) describes the winning submission to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. The authors investigate two complementary fine-tuning strategies for building an embedding model that matches vocal imitations to target sounds. The first applies contrastive learning on top of a frozen, pretrained CED (Contrastive Encoder-Derived) audio encoder. The second performs joint contrastive-triplet learning using a MobileNetV3 encoder, incorporating semi-hard negatives to improve discrimination between similar sound embeddings. The report covers the methods, training setups, and competition results, and has been updated after the challenge to include additional details released post-competition. This work demonstrates practical transfer-learning and metric-learning approaches for vocal imitation-based audio retrieval, a task relevant to sound effect search and audio search engines.

Paper Overview

Field: Machine Learning / Audio Retrieval
Authors: Aditya Bhattacharjee, Christos Plachouras, Sungkyun Chang, Emmanouil Benetos
Published: 2026-08-19
arXiv: 2608.19174

Abstract

This technical report describes our winning submission to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. We investigate two complementary fine-tuning strategies: contrastive learning with a frozen, pretrained CED encoder, and joint contrastive-triplet learning with semi-hard negatives using a MobileNetV3 encoder. This report has been updated for posterity to include details released after the challenge.

Key Points

Task: Vocal imitation-based query-by-sound — retrieving target sound effects from a user's vocal imitation of the sound.
Strategy 1: Contrastive fine-tuning built on a frozen, pretrained CED encoder, leveraging a strong general-purpose audio representation without updating the backbone.
Strategy 2: Joint contrastive-triplet learning with a lightweight MobileNetV3 encoder, using semi-hard negatives to sharpen the embedding space against confusable sounds.
Result: The combined approach won the AES AIMLA 2025 Challenge on this task.
The report was updated after the challenge to include details released post-competition.

Relevance

The work offers practical guidance for audio embedding fine-tuning — comparing parameter-efficient adaptation (frozen backbone + contrastive head) against full metric-learning pipelines (triplet loss with semi-hard negative mining) — applicable to sound effect search and content-based audio retrieval systems.