English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Towards Interpreting Any Residual Stream Direction by Scaling Activation Metamodels (MAEMs)

Forum topic · 小凯 · 2026-11-06

Summary

This arXiv paper (2611.08465) by De Schamphelaere et al. introduces Maximally-Activating Example Metamodels (MAEMs), language models trained to map residual stream directions in a subject model to text exemplars that strongly activate those directions. The training objective maximizes cosine similarity between a supplied residual stream direction and the activations produced by generated text, enabling label-free RL scaling across diverse direction sources. A Qwen3.6-27B MAEM outperforms corpus search and natural language autoencoders (NLAs) at inverting held-out activations, beats corpus search on most held-out languages/domains (scanned over 10M tokens), activates 89% of held-out SAE features, matches NLAs at identifying steering-vector concepts, more often recovers LoRA backdoor payloads than corpus search, and can verbalize some of the model's working-memory inner monologue. Unlike NLA explanations, MAEM exemplars are verifiable via forward passes on the subject model, making the objective a grounded alternative for scaling metamodel training.

Paper Overview

  • Field: Machine Learning
  • Authors: Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger, Tomáš Gavenčiak, Gavin Leech, Euan Ong, Jiahai Feng
  • Published: 2026-11-05
  • arXiv: 2611.08465
  • Abstract

    To interpret a direction in a language model's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to what that corpus contains. The authors develop Maximally-Activating Example Metamodels (MAEMs), language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions.

    The training objective is to maximize the cosine similarity between a supplied residual stream direction and the activations of the text generated by the MAEM. This label-free RL objective helps scale MAEMs across diverse sources of residual stream directions.

    Key Results

    A Qwen3.6-27B MAEM produces informative exemplars:

  • Inversion: Better at inverting held-out activations than corpus search or natural language autoencoders (NLAs)
  • Coverage: Beats corpus search (scanned over 10M tokens) on most held-out languages and domains
  • SAE features: Activates 89% of held-out SAE features
  • Steering vectors: Performs comparably to NLAs at identifying concepts behind steering vectors
  • Backdoors: More frequently recovers effective LoRA backdoor payloads than corpus search
  • Model introspection: Can articulate some of the model's working-memory "inner monologue"

Why It Matters

Unlike explanations generated by other metamodels such as NLAs, MAEM exemplars can be verified by forward passes on the subject model. This makes the MAEM objective a grounded, verifiable alternative for scaling metamodel training.

--- *Collected automatically on 2026-11-07*

Tags

#machine-learning#interpretability#residual-stream#reinforcement-learning#language-models#sparse-autoencoders#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636826