RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering
- Authors: Yang Bai, Christan Earl Grant, Daisy Zhe Wang
- Published: 2025-01-23
- Paper: https://arxiv.org/abs/2501.13297
- Code & data: https://github.com/TonyBY/RAMQA
- Cascade architectures (retrieval + re-ranking + generation) remain mainstream, while agentic paradigms increasingly treat retrieval strategy itself as learnable.
- High-quality instruction data is critical; synthetic data must guard against knowledge leakage and distribution shift.
- Offline metrics increasingly diverge from online user satisfaction; LLM-as-judge should be cross-validated with human evaluation.
- Latency, cost, interpretability, and safety are hard constraints for production deployment and cannot be ignored in favor of academic benchmarks alone.
- RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering. arXiv:2501.13297. https://arxiv.org/abs/2501.13297
Background and Motivation
Multi-modal retrieval-augmented Question Answering (MRAQA), which integrates text and images, has attracted significant attention in information retrieval (IR) and NLP. Traditional ranking methods rely on small encoder-based language models, which are incompatible with the modern decoder-based generative large language models (LLMs) that now drive many NLP tasks. RAMQA was proposed to bridge this gap by unifying ranking and generation within a single LLM-era pipeline.
Approach
RAMQA combines learning-to-rank methods with generative, permutation-enhanced ranking techniques:
1. Pointwise multi-modal ranking: A pointwise ranker is trained with LLaVA as the backbone to score multi-modal document candidates. 2. Generative re-ranking: A LLaMA model is instruction-tuned via an innovative autoregressive multi-task learning approach to re-rank the top-k documents. The generative ranking model outputs re-ranked document IDs and specific answers from document candidates in various permutations.
Results
Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, demonstrating the effectiveness of combining LLM-based generative ranking with multi-modal retrieval.
Context and Insights
RAMQA sits at the intersection of RAG and large-scale search systems, addressing how to redistribute the responsibilities of retrieval, ranking, and generation in the LLM era. Key observations from the surrounding discussion:
Limitations
Reported limitations common to this line of work include experiment scale constrained by GPU budgets, benchmarks that may not match real user distributions, English-centric data with unknown cross-lingual generalization, and safety risks when agents access the open web.
Original Abstract (verbatim)
> Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: https://github.com/TonyBY/RAMQA