When Embedding Models Meet "1 meter = 100 centimeters": They Don't Actually Know
A simple test that all 24 embedding models failed
Ask a language model: "Which is longer, 1 meter or 100 centimeters?" You'd expect "they're the same." But look inside the embedding space instead—how close are "1 meter" and "100 centimeters" as vectors? Strangely, they aren't close at all.
Even stranger: in the embedding space, "2.5 meters" is more similar to "7.5 meters" than to "3 meters." It's like a ruler where 2.5 cm sits next to 7.5 cm, while 2.5 cm and 3 cm are miles apart.
This is not an isolated bug. It is the core finding of the paper *Embedding Models Measure in Peculiar Ways* by Opitz and Andrianos (September 2026), after systematic testing of 24 mainstream embedding models: embedding models' understanding of physical measurement is broadly wrong—and wrong in systematic ways.
- Paper: https://arxiv.org/abs/2609.20821
- Code: https://github.com/flipz357/embed-and-measure
- Classic Sentence-Transformers baselines
- Contrastively trained BERT-style encoders
- ModernBERT-based encoders
- LLM-derived embedding models (including the latest Qwen3-Embedding)
- Checker pattern: at small value ranges, similarity matrices alternate. Integer–integer pairs (2 m vs 3 m) score higher; integer–float pairs (2 m vs 2.5 m) score lower. Models aren't reading magnitude—they're reading "does this number look like an integer?"
- The "nine" curse: Qwen3-Embedding judged "nine liters" dissimilar to *all* liter expressions—including "9 liters." Written as a word vs. digits, the same quantity becomes a stranger to itself, while "one liter" and "1 liter" sit close together.
- Conversion collapse: "1 meter" should be equivalent to 100 cm, 1000 mm, 0.001 km. Instead, similarity curves were nearly flat—the models couldn't tell whether 100 cm or 1000 cm is closer to 1 meter.
- Paper: https://arxiv.org/abs/2609.20821
- Code: https://github.com/flipz357/embed-and-measure
What they did
Physical measurement systems are among the oldest, most standardized semantic spaces: 1 meter is exactly 100 centimeters, 1000 millimeters, 0.001 kilometers—objective, unique, context-free equivalences. If embedding models truly captured "semantic similarity," this should be the easiest case.
The researchers tested 24 models covering:
The method: construct many phrase pairs like "X meters" / "Y centimeters," compute cosine similarities, and plot heatmaps. If models understood physical distance, the heatmaps would show smooth diagonal decay.
Checkerboards, stripes, and the "nine" curse
The actual heatmaps looked like abstract art.
Three unsettling findings
1. All 24 models failed. From old mpnet to the latest Qwen3-Embedding, from 300-dim to high-dim LLM-derived models, none faithfully represented physical distance relations. This consistency points to a structural flaw in the embedding mechanism itself, not model size or data volume. 2. Stronger models aren't better. Against the "scale fixes everything" narrative, model family, embedding dimension, and release date had no significant effect. A 2024 flagship fared as poorly as a 2020 baseline. 3. String similarity is the culprit. Linear probes showed embedding similarity is driven mainly by surface string overlap, not numeric/semantic content: "2.5 meters" resembles "7.5 meters" because they share ".5 meters." Recalibration via linear transforms also failed—you can't fix a representation that never encoded the right information.
Why this matters
One might object: embedding models were built for search, retrieval, and clustering, not unit conversion. But that's the point. If models can't capture the most objective, unambiguous, context-free semantic relation—1 meter = 100 centimeters—what is their "semantic similarity" actually measuring elsewhere?
Our RAG systems, search engines, recommendation systems, and semantic caches all rest on the assumption that embedding distances reflect real semantic relations. This study shows that assumption is false, at least in the physical measurement subspace. Worse, "bigger isn't better" implies a structural blind spot of the training objective itself: contrastive learning teaches models to distinguish different sentences, but never explicitly teaches numeric magnitude or unit equivalence.
A deeper metaphor
This study illustrates a broader pattern: systems perform well on what is easy to measure and quietly fail on what matters. Toxicity classifiers measure surface toxicity, not structural bias; benchmark scores measure multiple-choice accuracy, not real reasoning; embedding similarity measures string overlap, not physical equivalence.
The paper's value isn't a minor bug report—it uses the simplest, most irrefutable test case to reveal how universal and stubborn this gap is. Twenty-four models across five years, none passing. It's not a bug; it's a feature—one we've been pretending not to see.
Next time you rely on embedding similarity for retrieval, remember: your search engine thinks "2.5 meters" is more similar to "7.5 meters" than to "3 meters." Then ask yourself how much you trust its other "semantically similar" results.
---