LLM's Relational Deficit: Knowing All the Words but Not How They Connect

Forum topic · 小凯 · 2026-05-22

Summary

A paper by Moses Boudourides, 'A Multi-Source Framework for Relational Validation of Large Language Models Using Expert-Curated Encyclopedic Sources' (arXiv:2605.22636, cs.SI), finds that LLMs excel at identifying domain concepts but systematically fail to reproduce the relational structures between them. Comparing LLM-generated concept networks against expert-curated encyclopedias across ten specialized disciplines, the study identifies a 'relational deficit': LLMs reduce precise disciplinary relations (causal, compositional, oppositional) into generic associations. Performance degrades in layers—concept recall is reliable, relation accuracy is unreliable, and global graph structure matches expert networks weakly in everyday domains and approaches randomness in highly specialized fields, described as 'complete relational failure.' The author argues this stems from pretraining's dependence on token co-occurrence, which flattens scholarly debates into undifferentiated relatedness. The paper proposes a three-layer evaluation framework and notes that current benchmarks like MMLU only test concept recognition. Limitations include limited disciplinary coverage, unresolved expert consensus, and unreported model specifics.

Overview

| Item | Detail | |------|--------| | Paper | A Multi-Source Framework for Relational Validation of Large Language Models Using Expert-Curated Encyclopedic Sources | | Author | Moses Boudourides | | arXiv ID | 2605.22636 | | Published | May 21, 2026 | | Category | cs.SI (Social and Information Networks) | | Core finding | LLMs can identify domain concepts but systematically fail to reproduce the relational structure between them, with 'complete relational failure' in highly specialized domains. |

The Difference Between a Phone Book and a Social Network

A phone book tells you who exists. A social network graph tells you who relates to whom, and how. The paper asks: which one resembles LLM internal knowledge?

The author drew expert-curated encyclopedias—concept networks compiled by scholars over years, not open-edit wikis—from ten specialized disciplines including sociology, political science, and philosophy. He then prompted LLMs to generate concepts and their relations in the same fields, and compared the results.

Concepts Recognized, Relations Not

Task A — Concept identification: LLMs perform well, listing textbook terms like class, power, social structure, capital, and habitus.
Task B — Relation reproduction: LLMs systematically fail. Instead of discipline-specific relations (e.g., Bourdieu's forms of capital vs. Marx's ownership of means of production), they collapse everything into generic associations—'X relates to Y,' 'A influenced B.'

The paper calls this the 'relational deficit.'

The More Specialized, the Worse the Collapse

Everyday knowledge: relational reproduction is relatively good—co-occurrence patterns repeated millions of times are well learned.
Mid-specialization (e.g., mainstream sociology): the deficit emerges; errors are 'approximately correct but imprecise,' e.g., conflating Weber's bureaucracy with Foucault's discipline.
High specialization (sub-branches, internal debates): complete relational failure—generated networks show almost no overlap with expert-curated ones, resembling plausible-looking random combinations.

A Three-Layer Diagnostic Framework

1. Concept recall: Does the model know which concepts exist in a field? Largely reliable. 2. Relation precision: Can it identify the relation type between concepts (causal, compositional, taxonomic, oppositional, similar)? Very unreliable, especially in specialized fields. 3. Structural integrity: How well does the full generated graph match the expert graph globally? Weak match in ordinary domains, near-random in specialized ones.

Most existing benchmarks (MMLU, BigBench) operate only at layer one—testing whether a model knows a word, not whether it understands where the word sits in a discipline's structure.

Why It Matters

Pretraining learns conditional probabilities between tokens, not structural relations between concepts. 'X appears near Y' may mean X and Y are *opposed* in academic debate (e.g., structural functionalism vs. conflict theory), yet the model compresses this into a uniform association signal. The LLM knows concept names and neighbors, but not the 'grammar' connecting them—like holding a phone book and pretending it is a social network map.

Honest Limitations

Ten disciplines cannot cover all human knowledge; natural sciences, engineering, and medicine were not examined.
Expert relation judgments can genuinely diverge (Bourdieu vs. Marx on class and capital); the abstract does not detail the consensus-building process.
'Complete relational failure' may reflect low pretraining coverage rather than an inherent inability to represent relations—a different mechanism from over-trained spurious relations.
The paper does not specify which LLMs were tested; heterogeneity across model scales and training data may be masked.

Takeaway

Extending Feynman's distinction between knowing the name of something and knowing the thing itself: knowing a concept's name, knowing its definition, and knowing where it lives in a discipline's knowledge structure are three different things. LLMs excel at the first two. The third has not yet been learned.