Code2LoRA: Hypernetwork-Generated LoRA Adapters for Repository-Level Code Adaptation with Zero Inference Overhead

Forum topic · 小凯 · 2026-06-07

Summary

Code2LoRA is a method from University of Waterloo researchers (arXiv:2606.06492) that uses a hypernetwork to generate repository-specific LoRA adapters for code language models, eliminating the need for retrieval-augmented context at inference time. A frozen repository encoder (Qwen3-Embedding-0.6B) compresses a codebase into a representation vector; a trainable hypernetwork (~720M parameters) then produces LoRA A/B matrices injected into all attention and MLP projections of a frozen Qwen2.5-Coder-1.5B backbone. Two variants are proposed: Code2LoRA-Static for stable codebases, and Code2LoRA-Evo, which uses a GRU to incrementally update adapters from commit diffs, preserving evolutionary history. Evaluated on RepoPeftBench, a new 604-repository assertion-completion benchmark (40K static and 215K evolutionary training tasks), Code2LoRA-Static outperforms the strongest baseline (full fine-tuning + RAG) by 9.9 percentage points (66.2% vs 56.8% in-repo exact match), while Code2LoRA-Evo gains 5.2pp over a single shared LoRA under software evolution and even exceeds per-repository LoRA upper bounds. RAG baselines underperform the unadapted model in cross-repository and evolving settings, supporting the paper's thesis that parameterized knowledge injection is more reliable and cheaper than context injection.

Overview

Code2LoRA (arXiv:2606.06492) by Liliana Hotsko, Yinxi Li, Yuntian Deng, and Pengyu Nie (University of Waterloo) addresses a key weakness of code LLMs: they don't know repository-specific APIs, conventions, or internal utilities. Instead of injecting repository context at inference time, Code2LoRA trains a hypernetwork that learns to *generate* a LoRA adapter for any given repository — so inference runs with zero extra tokens.

Key points

Architecture: A frozen repository encoder compresses the codebase into a vector; a trainable hypernetwork (~720M static / ~745M evo) generates LoRA A/B matrices (rank 16) injected into all 7 projection types (Q, K, V, O, gate, up, down) of a frozen Qwen2.5-Coder-1.5B backbone.
Repository encoding: Files are chunked (512 tokens), encoded with frozen Qwen3-Embedding-0.6B, mean+max pooled per file, then aggregated via weighted average (weights = content distinctiveness × file size × path importance).
Two variants:
*Code2LoRA-Static*: one-shot adapter generated from a repository snapshot.
*Code2LoRA-Evo*: a GRU incrementally ingests encoded commit diffs and maintains a hidden-state "memory" of the repository's evolution, from which the hypernetwork regenerates updated LoRA weights.
Training: Two stages (static LoRA generation, then GRU updates), supervised with exact match / edit similarity / CodeBLEU on assertion-completion tasks. Single H100 80GB, ~1–2 days.

RepoPeftBench benchmark

604 repositories (409/51/52 train/val/test + 92 temporal OOD)
Static: 40K train / 12K test; Evolutionary: 215K train / 87K test (organized by commit order)
Task: assertion completion — non-test code serves as repository context; the model must predict test assertion suffixes
Evaluation: Cross-Repo (CR), In-Repo (IR), and Temporal OOD splits

Results

Static setting (exact match):

| Method | Cross-Repo EM | In-Repo EM | |---|---|---| | Pretrained | 45.7% | 46.8% | | RAG (k=3) | 39.7% | 42.1% | | FFT + RAG | 53.9% | 56.8% | | Per-repo LoRA (upper bound) | — | 64.0% | | Code2LoRA-Static | 63.8% | 66.2% |

Code2LoRA-Static beats the strongest baseline by +9.9pp and matches/exceeds the per-repo LoRA upper bound without per-repository training. Notably, RAG underperforms the unadapted model in cross-repo settings.

Evolution setting: all methods degrade (pretrained drops to 31.5% CR), RAG collapses to 23.6%, but Code2LoRA-Evo reaches 60.3% CR / 64.5% IR — +5.2pp over a single shared LoRA and above the per-repo LoRA bound (64.2%). On temporal OOD repositories, Evo leads with 74.1%.

Why it works

Parameterized knowledge beats context injection: stable weights instead of fluctuating retrieval quality, zero inference-time token overhead, and natural cross-repository generalization via the hypernetwork.
The hypernetwork learns "how to write adapters": recognizing API patterns, project abstractions, and naming conventions — analogous to a developer quickly grasping a new codebase's style.
GRU incremental updates keep adapters current at low cost (only diffs processed) while retaining evolutionary history.

Limitations

Evaluated only on Python and only on assertion completion; other languages/tasks untested.
Backbone is small (1.5B); hypernetwork scaling to 7B/14B backbones is unverified.
Temporal OOD targets are systematically shorter (median 7 vs 12–13 chars), inflating absolute numbers, though relative rankings hold.
Generated assertions may carry safety/licensing risks requiring standard mitigations.

Conclusion

Code2LoRA shows that hypernetwork-generated, parameterized repository knowledge can match or exceed per-repository fine-tuning at scale, adapt continuously to software evolution via GRU updates, and do so with zero inference overhead — a practical blueprint for AI coding assistants that quietly adapt to whichever repository you open.

References

Hotsko L, Li Y, Deng Y, Nie P. *Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution*. arXiv:2606.06492, 2026.
Zong Y, et al. *Per-Repository LoRA for Code Adaptation*. 2025.
Charakorn R, et al. *Text2LoRA: Hypernetwork for Task-Specific LoRA Generation*. 2025.
Jain N, et al. *RepoBench: Repository-Level Code Evaluation*. 2025.
Ren S, et al. *CodeBLEU: A Method for Automatic Evaluation of Code Synthesis*. 2020.