RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering

Forum topic · 小凯 · 2026-07-05

Summary

RAMQA is a unified framework for multi-modal retrieval-augmented question answering (MRAQA), proposed by Yang Bai, Christan Earl Grant, and Daisy Zhe Wang (arXiv:2501.13297, January 2025). Traditional ranking methods rely on small encoder-based language models that are incompatible with modern decoder-based generative LLMs. RAMQA bridges this gap by combining learning-to-rank methods with generative permutation-enhanced ranking. It first trains a pointwise multi-modal ranker using LLaVA as the backbone, then applies instruction tuning to train a LLaMA model for re-ranking the top-k documents via an autoregressive multi-task learning approach. The generative ranking model produces re-ranked document IDs and specific answers from candidate documents in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines. Code and data are available at https://github.com/TonyBY/RAMQA. The paper addresses efficiency, scalability, and user-intent understanding challenges in RAG systems, discussing open issues such as evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual and multi-modal expansion.

Overview

RAMQA (arXiv:2501.13297, published 2025-01-23) is a unified framework for multi-modal retrieval-augmented question answering (MRAQA), authored by Yang Bai, Christan Earl Grant, and Daisy Zhe Wang.

Background

MRAQA, which integrates text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks.

Method

To bridge this gap, RAMQA combines learning-to-rank methods with generative permutation-enhanced ranking techniques:

1. Pointwise multi-modal ranker: trained using LLaVA as the backbone. 2. Generative re-ranker: a LLaMA model is instruction-tuned to re-rank the top-k documents using an autoregressive multi-task learning approach. 3. Permutation-enhanced ranking: the generative ranking model outputs re-ranked document IDs and specific answers from document candidates in various permutations.

Results

Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, demonstrating the effectiveness of the unified retrieval-ranking-generation approach.

Broader Context

The work sits at the intersection of RAG and large-scale search/recommendation systems. Notable insights for the field include:

Cascaded retrieval + re-ranking + generation remains the mainstream architecture, but agentic paradigms are making retrieval strategy itself a learnable object.
High-quality instruction data matters as much as click/session logs; synthetic data must guard against knowledge leakage and distribution shift.
The gap between offline metrics and online satisfaction is widening; LLM-as-judge evaluation needs cross-validation with human assessment.
Latency, cost, explainability, and safety policies are hard constraints for industrial deployment.

Resources

Paper: https://arxiv.org/abs/2501.13297
Code and data: https://github.com/TonyBY/RAMQA

Related entries include Cross-Modal Retrieval surveys, Clotho-AQA, ColPali, EA-VTR, and Generative Cross-Modal Retrieval.