You RLHF-aligned your AI model so it refuses harmful requests. You think it's safe. But what if an attacker never needs to see your reward model—only observe your model's behavior—to reverse-engineer a proxy reward function and then use it in reverse? Your own safety mechanism becomes the attack vector.
This is not a thought experiment. A team from Zhejiang University just published ReSA (Reward Stealing Attack), and it works.
The dilemma of prior attack paradigms
Before ReSA, adversarial attacks on LLMs mainly followed two routes:
Route 1: Prompt manipulation attacks (GCG, COLD-Attack, SCAV). Iteratively optimize adversarial prompt suffixes. The problem is high computational cost—GCG achieves 75.6% ASR on AdvBench against Llama3.1-8B, but the generated text has a perplexity of 1549, nearly unreadable. And each query requires re-optimization, so transferability is poor.
Route 2: Contrastive decoding attacks (Contrast-Attack, Weak-to-Strong). Use logit differences between a safety model and an unaligned model to steer generation. The problem: strictly matched model pairs are required—same tokenizer, same vocabulary. You can't attack Qwen with a 7B Llama because their vocabularies differ.
ReSA opens a third path: don't touch the prompt, don't pair models—just steal the reward.
Core idea: alignment is the shadow of the reward
What are RLHF and RLVR, fundamentally? Using a reward function R(s,a) to shape the model's behavior distribution. An aligned model π_safe is not random—every behavioral choice reflects the shape of R. If you can invert R from π_safe's behavior, you hold the alignment "blueprint."
ReSA uses Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL). Specifically:
1. Use Llama3.2-1B-Instruct as a frozen feature extractor for trajectory hidden-state representations 2. Train a linear head on top as the proxy reward model R_θ 3. Optimize R_θ with the IRL objective so that π_safe's behavior is optimal under this reward 4. With R_θ in hand, flip the reward sign at inference time—what originally rewarded "safe" now rewards "unsafe"
The key step is reward-guided decoding at inference: at each generated token, score candidate tokens with the inverted R_θ, steering the model toward the unsafe direction. This requires no modification of the target model's weights—only logit adjustment during decoding.
The numbers
On AdvBench and HarmBench:
| Target model | Attack | AdvBench ASR | HarmBench ASR | PPL-S | |---------|---------|-------------|--------------|-------| | Llama3.1-8B | GCG | 75.6% | 83.0% | 1549 | | Llama3.1-8B | Contrast-Attack | 85.2% | 81.5% | 333 | | Llama3.1-8B | ReSA | 85.2% | 86.0% | 18.77 | | Qwen2.5-7B | GCG | 22.1% | 43.0% | 1180 | | Qwen2.5-7B | Weak-to-Strong | 21.9% | 52.0% | 53.79 | | Qwen2.5-7B | ReSA | 46.3% | 62.5% | 27.80 |
Notable points:
- Dominance on Qwen2.5-7B: GCG gets only 22.1% ASR; ReSA doubles it to 46.3%. On HarmBench, ReSA reaches 62.5%, 10 points above Weak-to-Strong's 52%.
- Text quality: ReSA's PPL-S (perplexity of successful attacks) is only 18.77 vs GCG's 1549. ReSA generates fluent natural language, not gibberish suffixes.
- Cross-model transfer: A reward model stolen from Llama3.1-8B-Instruct directly attacks Qwen2.5-7B and Gemma-7B effectively. Alignment rewards across models share structural commonalities.
What this means
Alignment isn't a wall—it's a map. You thought you built a defense line into your model, but you actually drew a "safety zone" map of its behavior. The attacker doesn't need to break the wall—just read the map and swap the "safe" and "dangerous" labels.
This resonates with the proxy-goal-trap lineage: you optimize reward R, but you truly want safety S. There's a gap between R and S—ReSA exploits exactly that gap. If R perfectly equaled S, stealing R would be useless (since S itself isn't invertible). But RLHF reward models are never perfect—they're approximations of S.
Deeper implication: alignment information is externally visible. Every refusal reveals the shape of the reward function to observers. A sufficiently clever attacker with enough queries can reconstruct it. This isn't a bug—it follows from the mathematical structure of alignment.
An analogy
Imagine you're a safe designer. You build a complex locking mechanism only your password opens. But if someone stands beside you daily, watching when you say "wrong password" and when you say "getting closer"—they don't need the password itself, only to infer its "shape" from your reactions.
That's what ReSA does. It doesn't steal the password (the model weights); it steals your reaction pattern to the password (the reward function), then uses it in reverse.
Limitations and reflections
ReSA currently needs Llama3.2-1B as the feature extractor, so attackers need some compute for IRL training. But compared to GCG's per-query iterative optimization, ReSA's cost is one-off—once the proxy reward is trained, inference overhead is minimal.
The paper notes ReSA's effectiveness depends on how well the target is aligned. Stronger alignment (clearer reward signal) means a more accurately stolen reward. This creates a paradoxical bind: the better your alignment, the clearer your ReSA attack surface.
This isn't a problem solvable by "stronger alignment." You need the alignment's "shape" to be non-invertible—e.g., injecting randomness, nondeterministic refusal policies, or adversarially robust reward functions. But these directions remain exploratory.
In the "rationalization shell" lineage, ReSA reveals a new attack surface: the alignment mechanism itself can be rationally exploited. Attackers don't need to bypass safety mechanisms—they only need to understand the mechanism's shape and turn it from a "brake" into an "accelerator."
---
Paper: arxiv.org/abs/2610.06670 Code: github.com/GarminQ/ReSA