Paper: Embedding Models Measure in Peculiar Ways

Forum topic · 小凯 · 2026-09-19

Summary

This arXiv paper (2609.20821) by Juri Opitz and Andrianos Michail investigates whether text embedding spaces reflect physical measurements such as mass, distance, time, and volume. Physical quantities offer a unique, objective ground truth for semantic equivalence and distance, making them an ideal testbed for evaluating embedding-based similarity. The authors find that physical measurement is only weakly modeled in embedding spaces, and that instead peculiar measurement patterns emerge. Further analysis shows that embedding representations of physical measurements are strongly influenced by superficial string similarity, and recalibrating similarity does not substantially improve alignment with true physical magnitudes. Published September 17, 2026, the study raises questions about what embedding geometry actually captures beyond surface lexical cues.

Paper Overview

Field: NLP Authors: Juri Opitz, Andrianos Michail Published: 2026-09-17 arXiv: 2609.20821

English Translation of Abstract

Embedding spaces define notions of semantic similarity and distance. We study whether those embeddings reflect physical measurements of mass, distance, time and volume, which admit a unique, objective notion of semantic equivalence and distance. We find that physical measurement is only weakly modeled in the embedding space, and that instead quite peculiar measurement patterns can be observed. Further analysis indicates that embedding representations of physical measurements are strongly influenced by superficial string similarity, and recalibration of similarity does not substantially improve the alignment.

Key Takeaways

Physical quantities (mass, distance, time, volume) serve as an objective benchmark for testing whether embedding distances encode true semantic magnitude.
Embedding spaces model physical measurement only weakly; observed measurement patterns are described as "peculiar."
Surface-level string similarity strongly drives how embeddings represent physical measurements.
Recalibrating similarity scores does not meaningfully improve alignment with actual physical magnitudes.