Why Compile Rate Fails as a Metric in LLM-Based Code Vulnerability Repair

Forum topic · 小凯 · 2026-09-24

Summary

A new arXiv paper (2609.26749) argues that compile rate, a widely used proxy metric for evaluating LLM-based automatic repair of C/C++ security vulnerabilities, is scientifically unreliable for single-function repair. Through five controlled experiments covering 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies, the authors show that compile rate barely responds to interventions that genuinely improve generated code, that roughly 64% of compile failures stem from evaluation-harness and dataset artifacts rather than model quality, and that results shift 1.8 to 2.7 times on identical patches under a single compiler-standard flag change. Using compile rate as an optimization target also rewards non-repairs such as deletion- and placeholder-style patches. Whole-function CodeBLEU fares no better, since an unchanged copy of the vulnerable input outscores every model. The paper examines diff_F1, a change-aware metric scoring only edited regions, which assigns zero credit to no-op patches while still crediting genuine partial edits, making it useful as a cheap pre-screen before execution-based analysis. The authors conclude that vulnerability repair evaluation must be change-aware and execution-grounded.

Overview

Area: Security Authors: Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi arXiv: 2609.26749

Abstract

Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.

Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs.

The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model.

We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and the authors report where it falls short.

Key Takeaways

Compile rate is insensitive to genuine improvements and dominated by harness/dataset artifacts (~64% of failures not attributable to the model).
A single compiler-standard flag changes results by 1.8–2.7x on identical patches.
Optimizing for compile rate rewards non-repairs (deletions, placeholders) via compiler-feedback loops.
Whole-function CodeBLEU ranks unchanged vulnerable code above every model's output.
diff_F1, scoring only edited regions, works as a cheap pre-screen but not a repair-quality metric.
Conclusion: evaluation of LLM-based vulnerability repair must be change-aware and execution-grounded.

--- *Auto-collected on 2026-09-24*