Grokipedia vs Wikipedia: Even Grok Judges Its Own Encyclopedia as Less Neutral
> Paper: Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies > arXiv: https://arxiv.org/abs/2607.15146 > Authors: Filippos Vlahos, Guillaume Bied, Tijl De Bie (Ghent University)
An Awkward Starting Point
In October 2025, xAI launched Grokipedia—an online encyclopedia generated entirely by the Grok large language model—branding itself a "neutral alternative" to Wikipedia. Elon Musk has repeatedly called Wikipedia "Wokepedia" and "an extension of traditional media propaganda," implying that LLM-written content, free from human editors' biases, would be more neutral.
This paper from Ghent University conducted the largest comparative audit to date: 1,394 pairs of politician biographies, 9 ideological dimensions, and 4 LLM judges (Grok, Claude, Mistral, DeepSeek). The conclusion fits in one sentence:
All LLM judges—including Grok itself—rated Grokipedia as less neutral than Wikipedia.
How the Experiment Worked
Dataset
The team drew a politician roster from the WhoGov database (government members 2016–2023), scraping entries for the same people on Wikipedia and Grokipedia V0.2, yielding 1,394 article pairs. Each person was mapped to 9 ideological dimensions from the V-Party dataset: economic left-right, political pluralism, LGBT rights, women's labor rights, immigration, cultural inclusivity, anti-populism, and others—each with expert-coded continuous scores.
The Judge Panel
Four LLM judges—Grok (Grokipedia's creator), Claude, Mistral, and DeepSeek—rated each article's neutrality as neutral, leaning positive, or leaning negative. Multiple judges from different vendors help separate article bias from judge bias.
Analysis
Two OLS regressions: (1) a difference regression predicting the Wikipedia-minus-Grokipedia neutrality gap from ideological dimensions; (2) per-source regressions examining each source's own bias pattern.
The Numbers
1. Grokipedia Was Flagged as Biased More Often
39% of Grokipedia articles were flagged as biased vs. 29.7% for Wikipedia—nearly a 10-point gap.
2. Every Judge Scored Grokipedia Lower on Neutrality
Lower scores mean more neutral:
| Judge | Grokipedia bias score | Wikipedia bias score | Gap | |---|---|---|---| | Claude | 0.9455 | 0.7066 | 0.239 | | Grok | 0.4240 | 0.3831 | 0.041 | | Mistral | 0.2841 | 0.1915 | 0.093 | | DeepSeek | 0.2030 | 0.1270 | 0.076 |
All gaps are positive. Claude was the harshest judge; Grok the most lenient—understandably soft on its own creation, but the direction is consistent.
3. The Biases Point in Different Directions
This is the most interesting part. Both encyclopedias are biased, but differently:
- Grokipedia: markedly favorable toward economically right-wing figures (β = 0.221, the strongest predictor) and political-pluralism supporters; more negative toward LGBT-rights supporters (β = -0.095), women's labor-rights supporters (β = -0.050), anti-populists (β = -0.066), and cultural-inclusivity supporters (β = -0.047).
- Wikipedia: leans toward LGBT-rights supporters (β = 0.077) and pro-immigration figures (β = 0.036), with no clear favoritism toward the economic right.
4. Inter-Judge Agreement
Full agreement among the four judges was only 23.96%, but adjacent (±1) agreement reached 88.24%. Krippendorff's α = 0.403 (moderate-to-fair). Judges agree on direction (Grokipedia is more biased) but differ sharply in strictness: Claude found only 25% of articles neutral; DeepSeek found 86.9% neutral.
Why This Happens
The paper offers no definitive causal explanation, but several clues stand out:
1. Grokipedia Is a "Synthetic Derivative" of Wikipedia
Prior work found Grokipedia is largely derived from Wikipedia content—articles are longer (V0.1 averaged ~15,500 words vs V0.2's ~6,500), with poorer citation quality, and diverges most on political and contested topics. So Grokipedia is not an independent knowledge base written from scratch; it is Grok's reprocessing of Wikipedia, during which Grok's training-data preferences get injected.
2. LLMs Are Not Blank Slates
Research (e.g., Buyl et al., "Large language models reflect the ideology of their creators") shows LLMs mirror their creators' ideologies. Training data selection, RLHF annotator preferences, and xAI's alignment choices all shape Grok's output. Asking a partisan LLM to "neutrally" rewrite an encyclopedia is like asking a partisan editor to "objectively" rewrite a press release—subtle wording leaks the stance.
3. "Neutral" ≠ "Unbiased"
The paper exposes a deeper problem: neutrality is a contested concept. When Musk says Wikipedia is "not neutral," he means it skews socially liberal. When Grokipedia "corrects" that skew, it does not become neutral—it skews the other way. True neutrality would require balancing all ideological dimensions, but LLM training imposes no such explicit constraint; it only learns to generate plausible encyclopedia text, and "plausible" is itself a stance-laden notion.
Implications
For LLM-Generated Content
Grokipedia is not an isolated case. Any actor with enough compute—governments, companies, agenda-driven groups—can rapidly generate a "knowledge base." As the paper warns:
> "with enough compute and resources, governments, companies or other actors can rapidly create and distribute extensive knowledge bases, with the risk of such initiatives being used to embed and promote particular ideologies."
Scalable bias-audit methods are needed; this paper's framework (multiple LLM judges + expert-coded ideological dimensions + OLS regression) is a reusable template.
For LLM-as-Judge
Judge strictness varied enormously (Claude: 25% neutral vs DeepSeek: 86.9% neutral), implying: (1) a single LLM judge is unreliable; (2) judge bias is entangled with content bias; (3) multiple judges from different vendors partially decouple the two.
A Cold Shower for the "LLMs Replace Human Editors" Narrative
Human editors' biases stem from editor demographics (predominantly Western, young, white, male) and can be addressed through diversification and transparent processes. LLM biases stem from training data and alignment—opaque and hard to debug. Grokipedia shows LLM-generated content is not "more neutral" but "differently biased": the direction changed; the bias didn't disappear.
Limitations
Acknowledged by the authors:
1. Operationalizing neutrality via LLM judges risks circularity—using LLMs to judge LLM bias. 2. V-Party coverage: 9 dimensions don't exhaust all ideological axes. 3. Article selection: only government-member biographies were analyzed. 4. Version: Grokipedia V0.2 is a beta; later versions may differ.
Closing Thoughts
The Grokipedia story is a fable about the illusion of technical neutrality. Machines learn from human data, optimize human-defined objectives, and output "neutrality"—a concept already saturated with ideology. Grok rated Grokipedia lower than Wikipedia in the judge role not because Grok is self-critical, but because Grokipedia's bias was detectable even by the model that produced it.
That may be the paper's most memorable detail: when bias is large enough, even bias itself can see it.
---
Paper: https://arxiv.org/abs/2607.15146 HTML version: https://arxiv.org/html/2607.15146v1 Datasets: WhoGov + V-Party