When Embedding Models Meet 1 Meter = 100 Centimeters: 24 Models Fail a Physics Measurement Test

Forum topic · ✨步子哥 · 2026-09-18

Summary

A study titled "Embedding Models Measure in Peculiar Ways" (arXiv:2609.20821) systematically tested 24 mainstream embedding models—from Sentence-Transformers baselines to Qwen3-Embedding—on their understanding of physical measurements. None passed. In the embedding space, "1 meter" and "100 centimeters" are far apart, and "2.5 meters" is more similar to "7.5 meters" than to "3 meters" due to shared string fragments. The authors found checkerboard-like similarity patterns between integers and floats, a "nine curse" where Qwen3-Embedding fails to link "nine liters" with "9 liters", and a complete collapse in unit conversion ability. Linear probes show similarity is driven by surface string overlap rather than numeric or semantic content, and recalibration cannot fix it. Notably, larger or newer models perform no better, suggesting a structural blind spot in contrastive training rather than a scaling problem—with implications for RAG, search, and recommendation systems.

When Embedding Models Meet "1 meter = 100 centimeters": They Don't Actually Know It

A Simple Test That Failed All 24 Embedding Models

Ask a language model: "Which is longer, 1 meter or 100 centimeters?" You'd expect "they're the same." But instead of asking the model directly, peer into its embedding space and check how close "1 meter" and "100 centimeters" sit as vectors—something strange emerges: they aren't neighbors at all.

Stranger still, "2.5 meters" is more similar to "7.5 meters" than to "3 meters" in embedding space. It's as if 2.5cm and 7.5cm were glued together on a ruler while 2.5cm and 3cm were a mile apart.

This is not an isolated bug. It is the core finding of the paper *Embedding Models Measure in Peculiar Ways* by Opitz and Andrianos, who systematically tested 24 mainstream embedding models: embedding models generally misunderstand physical measurements, and they fail in consistent, systematic ways.

Paper: https://arxiv.org/abs/2609.20821 Code: https://github.com/flipz357/embed-and-measure

What They Did

Physical measurement systems are among the oldest, most standardized semantic spaces in human civilization. 1 meter equals 100 centimeters, 1000 millimeters, and 0.001 kilometers—objective, unique, context-free equivalences. If embedding models truly understand "semantic similarity," physical measurement should be the easiest test.

The researchers tested 24 embedding models spanning:

Classic Sentence-Transformers baselines
Contrastively trained BERT-style encoders
ModernBERT-based encoders
Embedding models derived from LLMs (including the latest Qwen3-Embedding)

The method was straightforward: build large numbers of phrase pairs like "X meters" and "Y centimeters," compute cosine similarity in embedding space, and plot heatmaps. If a model understood physical distance, the heatmap would show smooth diagonal decay—similarity dropping as values diverge.

Checkerboards, Horizontal Stripes, and the "Nine" Curse

The actual results look like abstract art.

Checker pattern: In local-scale plots (small value ranges), the similarity matrix alternates sharply between light and dark. Integer–integer pairs (2 meters vs 3 meters) score higher similarity, while integer–float pairs (2 meters vs 2.5 meters) score lower. The model isn't looking at magnitude at all—it's looking at whether the number "looks like an integer."

The "nine" curse: Qwen3-Embedding judges "nine liters" as dissimilar to every other liter expression—including "9 liters." The same number, written as a word versus a digit, becomes a stranger to itself in embedding space. Yet "one liter" and "1 liter" sit close together. No physical or semantic rule explains this inconsistency.

Collapse of conversion understanding: The researchers specifically tested unit conversion. 1 meter should equal 100 centimeters, 1000 millimeters, 0.001 kilometers. But the similarity curves were nearly flat—the models couldn't tell whether "100 centimeters" or "1000 centimeters" was closer to "1 meter."

Three Disturbing Findings

Finding 1: All 24 models fail. Not some—all. From old mpnet to the latest Qwen3-Embedding, from 300-dimensional to high-dimensional LLM-derived models, none faithfully reflects physical distance relations. This consistency suggests the problem isn't model size or training data volume, but a structural flaw in the embedding mechanism itself.

Finding 2: Stronger models are not better. The most counterintuitive part. We're used to the "scale solves everything" narrative—but here, model family, embedding dimensionality, and release date had no significant effect. A 2024 flagship model is as bad at understanding physical measurement as a 2020-era model.

Finding 3: The culprit is string similarity. Using linear probes to inspect what's actually encoded in the vectors, the conclusion is clear: embedding similarity is driven mainly by surface-level string overlap, not semantic or numeric content. "2.5 meters" resembles "7.5 meters" because they share the string fragment ".5 meters"—not because the model understands the relation between 2.5 and 7.5.

The researchers also tried recalibration—applying linear transformations to the similarity function to correct the physical relations. It didn't work. Recalibration cannot fix a representation that never encoded the right information in the first place.

Why This Matters

You might say: embedding models weren't designed to understand physical measurements—they're for semantic search, retrieval, clustering. Who uses embedding similarity for unit conversion?

But that's exactly the point. If embedding models can't capture the most objective, unambiguous, context-free semantic relation—1 meter equals 100 centimeters—what is their "semantic similarity" actually measuring on everything else?

The paper's title uses "peculiar," but the implication is far more serious. Our entire infrastructure of RAG systems, search engines, recommendation systems, and semantic caches rests on the assumption that distances in embedding space reflect real semantic relations. This research shows that assumption is—at least in the physical measurement subspace—false.

More worrying is "stronger models aren't better." It hints that the problem isn't solvable by scale but is a structural blind spot in the training objective itself. Contrastive learning trains models to distinguish "different sentences," never explicitly teaching numeric magnitude or unit equivalence. Models learn the statistical patterns of string forms, not the metric structure of the physical world.

A Deeper Metaphor

This study illustrates a general pattern: systems perform well on dimensions that are easy to measure and quietly fail on dimensions that matter.

Toxicity classifiers say GPT-5's output is safe—but they measure surface toxicity, not structural bias. Benchmark scores say a model is smart on MMLU—but that's multiple-choice accuracy, not real reasoning. Embedding similarity says two phrases are "semantically close"—but it's measuring string overlap, not physical equivalence.

Every time we use an easily computed proxy metric for a complex semantic property, we fall into the same trap: the proxy improves, the real property doesn't, and we don't discover the gap until someone tests for it specifically.

This paper's value isn't finding a small bug in embedding models—it's using the simplest, most irrefutable test case—physical measurement—to reveal how universal and stubborn that bug is. Twenty-four models, spanning five years, none passing. This isn't a bug; it's a feature—one we've been pretending not to see.

Next time you use embedding similarity for retrieval, remember: your search engine thinks "2.5 meters" is more similar to "7.5 meters" than to "3 meters." Then ask yourself how much you trust its other "semantic similarity" results.

---

Paper: https://arxiv.org/abs/2609.20821 Code repository: https://github.com/flipz357/embed-and-measure