Steering Code LLMs with Activation Directions for Language and Library Control

Forum topic · 小凯 · 2026-03-27

Summary

Code large language models (LLMs) often default to specific programming languages and libraries when given neutral prompts. This paper investigates whether such preferences are encoded as approximately linear directions in the models' activation space that can be manipulated at inference time. The authors, Md Mahbubur Rahman, Arjun Guha, and Harshitha Menon, use a difference-in-means method to estimate layer-wise steering vectors for five language/library pairs, then add these vectors to the model's hidden states during generation to control which language or library the model uses. The work, posted on arXiv (2603.23629), suggests that language and library choices in code generation can be steered without fine-tuning, offering a lightweight inference-time control mechanism. This has implications for developers who need code LLMs to target specific technology stacks, and for understanding how preferences are internally represented in these models.

Paper Overview

Field: Machine Learning
Authors: Md Mahbubur Rahman, Arjun Guha, Harshitha Menon
Published: 2026-03-26
arXiv: 2603.23629

Abstract

Code LLMs often default to particular programming languages and libraries under neutral prompts. The authors investigate whether these preferences are encoded as approximately linear directions in activation space that can be manipulated at inference time.

Using a difference-in-means method, the study estimates layer-wise steering vectors for five language/library pairs and adds them to the model's hidden states during generation, steering the model toward desired languages or libraries without retraining.

Key Points

Code LLMs exhibit systematic default preferences for programming languages and libraries.
These preferences appear to be encoded as approximately linear directions in activation space.
Steering vectors are computed with a difference-in-means approach at each layer.
Adding these vectors to hidden states at inference time controls language/library choice for five language/library pairs.

---

*Auto-collected on 2026-03-27.*