Understanding the Prompt Sensitivity
arXiv: 2604.18389 | 2026
Have you ever encountered this strange situation? You ask ChatGPT to "explain quantum entanglement" and get a great answer. Then you rephrase: "Can you tell me what quantum entanglement is all about?" — and receive a completely different, even contradictory response. The meaning is identical, yet the answers differ wildly.
This is not your imagination. The phenomenon is known as Prompt Sensitivity, one of the most frustrating problems in current LLMs. A research team from Kyoto University has offered a mathematical explanation using an unexpected tool: Taylor expansion.
An Insight from Image Classification
The team first trained a ResNet-101 on CIFAR-10 image classification and observed that in intermediate layers, representations of same-class samples increasingly cluster together. Two photos of cats taken from different angles look very different at the input layer, but their internal representations converge as depth increases. Intuitively, a classifier must "group similar things together" to classify accurately.
Do LLMs behave this way?
The Answer: No
When the same analysis was applied to LLMs, the result was surprising: LLMs do not cluster semantically similar prompts — they disperse them. Given 12 different phrasings of the same question, these semantically equivalent prompts end up in different internal locations, leading to highly variable outputs.
What the Taylor Expansion Reveals
The team treated the LLM as a multivariate function and applied a first-order Taylor expansion, using the Cauchy-Schwarz inequality to derive an upper bound on output differences between semantically identical prompts. The bound is determined by two factors:
1. The distance between the two prompts (their difference in input space) 2. The norm of the model's gradient (how sensitive the model is to input changes)
The key finding: although LLM gradients shrink in deeper layers (which helps), hidden representation dimensionality increases, and same-class samples are not clustered. Combined, the upper bound keeps growing with depth and never converges low enough.
In plain terms: the internal structure of LLMs makes it impossible to "ignore" differences between semantically identical but differently worded inputs. Small per-layer differences accumulate, producing vastly different outputs.
Additional Findings
- Prompt templates matter more than the question itself. Analyzing logit variance showed that the prompt "frame" (e.g., "Please answer the following question" vs. "Based on your knowledge, answer") influences output even more than the question content.
- Different types of edits have different impacts. Modifying the first half of a prompt (near the question) typically matters more than modifying the second half; slight token misalignment triggers more sensitivity than synonym substitution.
- The derived bound correlates strongly with PromptSensiScore, meaning the theoretical tool can practically predict how prompt-sensitive an LLM is.
Can It Be Fixed?
The team tested Activation Steering: since the problem is that semantically identical prompts get dispersed internally, the intervention directly modifies hidden states at a target layer so that semantically identical prompts produce closer representations. Experiments confirm this effectively reduces prompt sensitivity, validating the Taylor expansion analysis.
Why It Matters
The study's value lies in providing a mathematical framework for understanding a fundamental LLM flaw. Prompt sensitivity was previously treated as an empirical phenomenon. This research shows it is not a bug but an inherent property of LLM architecture: the Transformer design prevents clustering of similar inputs the way traditional classifiers do — and this is the root cause of prompt sensitivity.
This has important implications for LLM design and training. Perhaps future architectures need an explicit "semantic invariance" mechanism, teaching models to ignore phrasing differences and focus on meaning itself.
---
Paper: arxiv.org/abs/2604.18389 Code: github.com/ku-nlp/Understanding_the_Prompt_Sensitivity