The r=0.85 Discovery
In 2025, Ray Dalio — founder of Bridgewater Associates — said on CNBC: "there will absolutely be a debt crisis."
If you are a computational social science researcher, you might ask: is "absolutely" genuine epistemic certainty, or a rhetorical habit? Do pessimists use "definitely" to strengthen their arguments, or "possibly" to hedge their reputational risk?
This is a question that can be answered with data. You collect a decade of Dalio's interviews, score each sentence on two dimensions — negative affect word frequency and emphatic certainty word frequency — and compute the correlation.
The result: r = 0.851, p < 0.001. A stunningly clean finding — the more pessimistic Dalio is, the more certain he sounds. This matches the popular intuition about "doomsayers": pessimists amplify fear with certainty, while optimists hedge to manage their reputations.
You extend this to three more public figures: Cathie Wood (tech optimist), Kenneth Rogoff (academic economist), Peter Zeihan (geopolitical determinist). All four show the same pattern, with r ranging from 0.725 to 0.932, all p < 0.01.
This is nearly a perfect computational social science finding: large sample (85 interviews, 32,625 sentences), cross-speaker consistency, large effect size, high statistical significance. Top-conference material.
But the finding may be fake.
Not data fabrication, not p-hacking, not sampling bias. Something more insidious: the measurement tool itself manufactured the correlation.
The Blind Spots of Keyword Lexicons
Bo Chen's 2026 paper *When Certainty Is an Artifact* (Institute of Computing Technology, Chinese Academy of Sciences) did something simple: scored the same corpus with two methods.
Method 1: Keyword lexicon. Count emphatic words ("always", "clearly", "absolutely", "certainly") as the certainty score; count negative words ("never", "no", "not") as the negative-affect score. This has been the field's standard approach for twenty years.
Method 2: LLM zero-shot semantic classification. Use a large language model to judge each sentence at the semantic level — accounting for negation, modification, polysemy, and context — and score whether it semantically expresses certainty/negativity.
Results on the same corpus:
| Speaker | Keyword r(neg, emphatic) | LLM r(neg, emphatic) | |---------|--------------------------|----------------------| | Ray Dalio | 0.851 (p<0.001) | 0.206 | | Cathie Wood | 0.725 (p=0.003) | negative | | Kenneth Rogoff | 0.917 (p<0.001) | negative | | Peter Zeihan | 0.932 (p<0.001) | no significant correlation |
r=0.851 became r=0.206. For three of four speakers, the correlation vanished or reversed.
This is not simply "Method 2 is more precise than Method 1." This is the same dataset, the same research question, two measurement tools — completely opposite conclusions.
Why Lexicons Get It Wrong
The paper identifies five systematic blind spots in keyword lexicons, each pushing the certainty measurement away from ground truth:
Blind spot 1: Negation. "I am never absolutely totally confident" — this sentence contains two emphatic words, "absolutely" and "totally", so the lexicon scores it as highly certain. But semantically, "never" negates the entire expression: this is a low-certainty statement. The lexicon cannot see "never" negating "absolutely".
Blind spot 2: Polysemy. "a certain body" — "certain" here means "some", not "sure". The lexicon counts it as a certainty word. English is full of such polysemous words.
Blind spot 3: Modification. "sort of clear" — "clear" is a certainty word, but "sort of" hedges it. The lexicon sees "clear" and not "sort of".
Blind spot 4: Framing. "I think for everyone" — "I think" is a hedging frame, but "everyone" is an intensifier. The lexicon only counts "everyone".
Blind spot 5: Quantification. "almost everything" — "everything" is a universal quantifier (emphatic), but "almost" weakens it into a hedge. The lexicon only counts "everything".
Each blind spot misclassifies "actually hedged" statements as "emphatic" ones. And negative discourse naturally attracts such structures — people delivering bad news habitually say things like "never absolutely", "sort of clear that it's bad", "almost everything is wrong" — so the keyword method manufactures a spurious "negative ↔ emphatic" correlation.
The correlation is not about the speakers' psychology; it is about the statistical structure of English vocabulary.
The Deeper Problem: A Category Error
The paper's sharpest argument is not "keyword methods are inaccurate", but:
> Treating keyword counts as measurements of epistemic certainty is a category error.
A category error, in philosopher Gilbert Ryle's sense, is treating a concept as belonging to the wrong category. The classic example: after touring the library, lab, and classrooms, asking "but where is the university?" — the university is not another thing alongside the library; it is the way these institutions are organized.
The category error here: keyword lexicons actually measure "negative discourse naturally attracts emphatic vocabulary" — a statistical regularity of English — not the speaker's epistemic state. The two get conflated.
The paper offers crucial evidence: four speakers with wildly different personalities (optimist Wood, academic Rogoff, geopolitical Zeihan, doomsayer Dalio) all show r=0.72–0.93 under the keyword method. If the correlation reflected psychology, four opposite personalities should not share the same mental state. But if it reflects a lexical statistical regularity — then of course everyone looks the same, because everyone speaks English.
A finding about psychology was actually a finding about English.
What the LLM Revealed
If the LLM method had merely "refuted" the keyword finding, the paper's value would be limited — just a falsification. But the LLM method uncovered a pattern the lexicon could not see:
A strong correlation between negative affect ↔ hedging language.
| Speaker | LLM r(neg, hedged) | |---------|---------------------| | Kenneth Rogoff | 0.875 (p=0.001) | | Peter Zeihan | 0.722 (p=0.008) |
This matches intuition: pessimists hedge ("maybe", "probably", "not entirely sure"), rather than emphasize. The pattern is especially pronounced for Rogoff — academic training makes him wrap pessimistic predictions in hedges.
The keyword method cannot see this because hedge-word frequencies ("maybe", "perhaps", "sort of") are drowned out by emphatic-word frequencies. An LLM can distinguish "sort of clear" as a hedge rather than emphasis; a lexicon cannot.
Broader Implications
On its surface the paper discusses a specific measurement problem — keyword lexicons vs LLM classification. But its implications go far beyond methodology:
First, computational social science's reproducibility crisis may be deeper than we think. Many published "findings" may not be about human behavior at all, but about measurement tools. When a more precise tool re-measures and the effect disappears, this is not a replication failure — the original study measured the wrong thing.
Second, LLMs are not just "better classifiers" — they are "different instruments". The difference between keyword and LLM methods is not precision but category — they do not measure the same thing at all. It is like measuring temperature with a ruler versus a thermometer: not a precision issue, a question of what is being measured.
Third, "statistically significant + large effect size" does not equal "real finding". r=0.851, p<0.001, N=21 — textbook-level strong evidence. Yet it is entirely an artifact of the measurement tool. Statistical significance only tells you an effect is not random noise; it cannot tell you the effect is the one you think it is.
A Question for the Reader
The paper does not conclude with the simple prescription "replace lexicons with LLMs". LLMs have their own problems: their judgments are black-box, unexplainable, and may differ across models (the paper ran cross-model robustness checks with largely consistent results, but this guarantees nothing about the future).
The deeper question: how do we know any measurement tool measures what we think it measures?
The lexicon method took twenty years to be revealed as mismeasuring. Could the LLM method be similarly exposed twenty years from now? Is it possible that the construct "epistemic certainty" itself lacks a valid operational definition, and every measurement tool only captures one of its projections?
This paper cannot answer that question. But it raises it sharply — with a concrete r=0.851 → r=0.206 example you cannot ignore.
Next time you see a computational social science finding that is "statistically significant, large effect, consistent across samples", ask first: is this finding about people, or about the measurement tool?
---
Paper link: When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance Author: Bo Chen (Institute of Computing Technology, Chinese Academy of Sciences)