English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering (Jan 2025, arXiv)

Forum topic · 小凯 · 2026-07-05

Summary

CoReQA is an academic paper published on arXiv in January 2025 (arXiv:2501.03447) that investigates the capabilities of large language models (LLMs) in code repository question answering. Authored by Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, and colleagues, the work sits at the intersection of information retrieval, code intelligence, and LLM-based search systems. It addresses how well language models can answer questions grounded in real-world code repositories, a task that requires understanding repository structure, cross-file dependencies, and developer intent. The paper contributes a benchmark and evaluation protocol designed to expose the strengths and limitations of current LLMs on repository-level QA, positioning itself within the broader shift from traditional pipeline-based retrieval toward retrieval-augmented and agentic search paradigms. According to the forum post, key discussion themes include retriever and reranker design, learning strategies such as supervised fine-tuning and reinforcement learning, evaluation metrics like Recall@k and task success rate, and open challenges including hallucination, latency, cost, and benchmark fidelity to real user distributions. Readers interested in code intelligence, RAG systems, or LLM evaluation should consult the original arXiv PDF for precise experimental numbers.

CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering (Jan 2025, arXiv)

  • Paper: CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering
  • Authors: Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, et al. (9 authors total)
  • Published: January 2025
  • Link: https://arxiv.org/abs/2501.03447
  • Type: Academic paper
  • Topics: Question Answering, Search Engine Evaluation, Information Retrieval

    • Overview

    CoReQA is a January 2025 arXiv paper (arXiv:2501.03447) that studies how well large language models perform question answering over code repositories. Repository-level QA is a demanding task: unlike isolated function-level code completion, it requires models to reason about repository structure, cross-file dependencies, and the developer's intent behind a natural-language question.

    Context and Motivation

    The work is situated at the intersection of information retrieval and code intelligence. Traditional pipeline approaches that separate retrieval, ranking, and generation struggle to meet modern requirements for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. CoReQA aims to systematically probe the theoretical and practical boundaries of LLMs in this setting, and to provide a comparable framework for evaluating repository question answering.

    Key Points (based on available metadata)

  • The paper proposes a benchmark/evaluation protocol for code repository question answering with LLMs.
  • It relates to broader trends in retrieval-augmented generation (RAG) and agentic search, where retrieval steps and tool use become part of the model's decision space.
  • Relevant methodological components discussed in this research area include dense/sparse retrieval representations, rerankers, planners, and generation with feedback mechanisms.
  • Typical evaluation considerations include Recall@k, task success rate, latency, and token cost, with baselines such as BM25, dense retrieval, cross-encoder reranking, and retrieval-free LLMs.
  • Open challenges highlighted in this space include benchmark fidelity to real user distributions, hallucination, latency and cost constraints, and cross-lingual generalization.

    • Caveats

    This post is based on the paper's abstract and public metadata. Exact benchmark composition, dataset sources, and quantitative results are not reproduced here — readers citing numerical findings should verify against the original PDF at https://arxiv.org/abs/2501.03447.

    Who Should Read This

  • Researchers: Reproduce core comparisons and check whether statistical significance and compute costs are reported.
  • Engineers: Assess how repository-level QA modules could plug into existing code search and developer-assistant stacks.
  • Product managers: Focus on user-perceivable benefits such as answer trustworthiness and multi-turn consistency rather than offline metrics alone.

Tags

#code-repository-question-answering#large-language-models#information-retrieval#rag#benchmark#arxiv#code-intelligence#llm-evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208820