CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering (arXiv 2501.03447)
This forum post on zhichai.net is an index entry for the January 2025 arXiv paper CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering (arXiv:2501.03447).
Paper metadata
| Field | Value | |-------|-------| | Title | CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering | | Authors | Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, et al. (9 authors total) | | Published | January 2025 (arXiv preprint) | | Link | <https://arxiv.org/abs/2501.03447> | | Category | Question Answering; Evaluation of Search Engines |
What the paper is about
CoReQA addresses question answering over code repositories — a setting where a language model must answer questions whose correct answers depend on understanding an entire codebase (cross-file context, project structure, and repository-level dependencies), rather than a single code snippet.
Based on the title and metadata, the work:
- Introduces a benchmark/dataset for repository-level code question answering;
- Evaluates the potential of existing language models on this task;
- Aims to expose gaps between current LLM capabilities and practical repository comprehension needs.
- Neural IR evolution: BM25 → dense retrieval → cross-encoder reranking → generative retrieval and LLM agents;
- RAG and agentic search, where retrieval becomes an iterative, plannable process;
- Evaluation shifting from static ranking metrics (nDCG, MRR) toward task success rates and answer faithfulness.
- Assessing The Potential Of Mid-Sized Language Models For Clinical QA (arXiv 2404.15894)
- Harnessing Large Language Models for Knowledge Graph Question Answering (arXiv 2412.18537)
- LLM-MedQA: Enhancing Medical Question Answering (arXiv 2501.05464)
- A Comprehensive Survey on Reinforcement Learning-based Agentic Search (arXiv 2510.16724)
- Original paper: CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering, arXiv, January 2025. <https://arxiv.org/abs/2501.03447>
Context within the field
Repository-level QA sits at the intersection of code intelligence and information retrieval, and is closely related to retrieval-augmented generation (RAG) for software engineering. The forum post situates CoReQA along a research line that includes:
Related entries cross-referenced in the post
Caveats
The forum post is largely a template-based index entry: several sections (abstract quote, experimental numbers) are placeholders. Quantitative results, benchmark construction details, and model comparisons should be verified against the original PDF at arXiv:2501.03447 before citation.