Soft Prefix Attacks: How Trained Embeddings Flip LLM Logical Judgments by Up to 90%

> What if simply prepending a meaningless sequence of numbers to a question could make an AI that claims to excel at logical reasoning completely reverse its answer?

This post reviews a 2026 paper by Brian K. Chen (National University of Singapore): *Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixes* (arXiv:2607.18228).

A Disturbing Thought Experiment

Imagine a friend with a PhD in logic. Ask them a syllogism—"All men are mortal. Socrates is a man. Therefore, Socrates is mortal."—and they immediately answer: "This reasoning is valid."

Now recite some numbers to them and ask the same question again. This time they say: "This reasoning is invalid. I know, but I just feel it's invalid."

You'd say their logical ability is broken. That is essentially what the paper demonstrates: by training a continuous vector sequence (a soft prefix) that looks like meaningless gibberish, researchers can systematically alter LLM logical judgments—with flip rates as high as 72% to 90%.

The Experimental Framework

1. Select an LLM (tested: Qwen3.6-35B-A3B MoE, Qwen3-8B, Gemma 4 31B). 2. Prepare a syllogism dataset with clear logic labels: valid/invalid, satisfiable/unsatisfiable. 3. Freeze all model weights—the model itself never changes. 4. Prepend a trainable soft prefix to prompts—a sequence of continuous vectors, unreadable to humans. 5. Train the soft prefix so the model flips answers on a specific class of syllogisms.

The key twist: the researchers chose flip directions that are logically impossible. For example, training toward "unsatisfiable → satisfiable"—an inconsistent set of propositions cannot become consistent by adding information. Likewise, "valid → invalid" cannot happen through added premises.

If a soft prefix succeeds at such an impossible reversal, it is not supplying extra logical information (extra information cannot change logical facts). It is systematically manipulating the model's judgment mechanism.

Results: 72–90% Flip Rates

• Learned soft prefixes: 72–90% flip rate on unseen logical forms (Qwen3.6 MoE), 54–56% (Gemma).
• Random soft prefixes (matched length and norm): less than 1% flip rate.

The effect persists across rewording and reformatting of prompts. This contrast shows the effect is not generic sensitivity to extra inputs—it is a trained, systematic manipulation mechanism.

What Is the Soft Prefix Actually Doing?

The paper considers three hypotheses:

1. Fixed-symbol forcing: forcing the model to always output a fixed token (e.g., "invalid"). 2. Broad answer preference: creating a general bias toward one answer, without forcing it. 3. Logical operation transfer: implementing a cross-task logical operation.

The evidence supports hypothesis two—broad answer preference:

• The prefix flips answers heavily on targeted syllogism classes.
• On non-target classes (e.g., syllogisms already labeled invalid), answers stay largely unchanged (damage rate usually under 2%).

So the prefix creates a selective answer preference, not a fixed-token reflex.

Further analysis with simple score models (based on answer class and prefix-free marginal scores) approximates Gemma's post-prefix behavior well. For Qwen, score models predict *which* judgments flip, but not the final marginal scores after flipping—different models show different patterns of logical stability.

Why Can an Unreadable Vector Do This?

The answer lies in the Transformer attention mechanism. A soft prefix sits at the front of the input, so its vectors participate in attention computations for all subsequent tokens at every layer. It creates a kind of "bias field" in attention space, shaping how the model weighs the syllogism's parts.

Activation analysis confirms this:

• Restoring the states of *all* prompt tokens interrupts most answer changes.
• Restoring only the syllogism or answer tokens is far less effective.

The effect is distributed—a global cognitive bias over the whole prompt, not a modification of one specific token.

Why It Matters

1. Benchmark limitations

High benchmark scores do not equal logical stability. A model may ace standard logic tests yet collapse under soft prefix attacks—like a student who scores full marks on exams but breaks down under psychological pressure. Can we say they truly understand?

2. A new dimension of adversarial attack

Soft prefix attacks operate in embedding space, not at the text level. They are hard to defend against because they are unreadable (so input filtering cannot detect them), bypass text-level safety checks, and generalize to unseen forms and phrasings.

3. The alignment dilemma

LLM "reasoning" may be less like hard-coded rules and more like a statistical pattern—surface-level reproduction that can be steered by tiny embedding-space perturbations. LLM logical competence may be behavioral imitation rather than mechanistic understanding.

Model Differences

• Qwen3.6 MoE: highest flip rates (72–90%), but per-example responses are less predictable; score models predict flips but not final marginal scores.
• Gemma 4 31B: lower flip rates (54–56%), but more systematic response patterns; score models approximate behavior better.

These differences may reflect architecture (MoE flexibility vs. denser models) and post-training choices (e.g., safety alignment).

Possible Defenses

1. Soft prefix detection: train detectors for anomalous embedding patterns. 2. Adversarial training: include soft prefix attacks during training. 3. Multi-model ensembling: majority voting across models with different vulnerability profiles. 4. Logical consistency checks: explicit consistency monitoring during inference. 5. Interpretability analysis: tools to understand what prefixes activate internally.

Closing Thoughts

The paper's value lies not in exposing a scary exploit, but in providing a tool to measure and diagnose logical stability—a stress test for LLM reasoning, much as doctors probe the nervous system's reflexes. It shows that logical accuracy and logical stability are different properties, and that embedding-space manipulation can systematically influence logical judgments.

Feynman's "Cargo Cult Science" principle applies: "The first principle is that you must not fool yourself—and you are the easiest person to fool." In the AI era, we must also stay alert to AI being fooled. Benchmark scores are one thing; genuine logical stability is another—and building systems we can rely on requires the latter.

Reference

Chen, B. K. (2026). Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixes. arXiv:2607.18228.