Grokipedia vs Wikipedia: Even Grok's Own LLM Judges Rate the AI Encyclopedia as Less Neutral
> Paper: Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies > arXiv: 2607.15146 > Authors: Filippos Vlahos, Guillaume Bied, Tijl De Bie (Ghent University)
An Awkward Beginning
In October 2025, xAI launched Grokipedia — an online encyclopedia generated entirely by the Grok large language model. It launched with a bold mission: to serve as a "neutral alternative" to Wikipedia. Elon Musk has repeatedly called Wikipedia "Wokepedia" and "an extension of traditional media propaganda." The implicit pitch: written by an LLM with no human editors' bias, it should surely be more neutral.
This paper from Ghent University conducted the largest audit to date: 1,394 pairs of politician biographies, 9 ideological dimensions, and 4 LLM judges (Grok, Claude, Mistral, DeepSeek). The conclusion fits in one sentence —
All LLM judges, including Grok itself, found Grokipedia less neutral than Wikipedia.
This isn't a paper written by Wikipedia's defenders — it's the result of four LLMs scoring each other. Grok rated its own offspring lower than the other judges rated it.
How the Experiment Worked
Dataset
The team extracted a list of politicians from the WhoGov database (government members of countries, 2016–2023), then scraped their articles from both Wikipedia and Grokipedia V0.2, yielding 1,394 article pairs — the same person, two sources.
Politicians were mapped to 9 ideological dimensions from the V-Party dataset: economic left/right, political pluralism, LGBT rights, women's labor rights, immigration, cultural inclusiveness, anti-populism, and more. Each dimension has expert-coded continuous scores.
The Judge Panel
Four LLM judges: Grok (Grokipedia's creator), Claude, Mistral, and DeepSeek. Each judge rated every article's neutrality as "neutral," "positive-leaning," or "negative-leaning." Using four models from different sources helps partially separate article bias from judge bias.
Analysis Method
Two OLS regressions:
1. Difference regression: the neutrality score difference between Wikipedia and Grokipedia, predicted by ideological dimensions. A positive coefficient means Wikipedia is more positive toward that ideology. 2. Per-source regression: regressions run separately for each source to reveal each one's ideological slant pattern.
What the Numbers Say
1. Grokipedia Was Flagged as Biased More Often
39% of Grokipedia articles were flagged as biased, versus 29.7% for Wikipedia — a gap of nearly 10 percentage points.
2. Every Judge Gave Grokipedia Lower Neutrality Scores
| Judge | Grokipedia bias score | Wikipedia bias score | Difference | |-------|-----------------------|----------------------|------------| | Claude | 0.9455 | 0.7066 | 0.239 | | Grok | 0.4240 | 0.3831 | 0.041 | | Mistral | 0.2841 | 0.1915 | 0.093 | | DeepSeek | 0.2030 | 0.1270 | 0.076 |
Lower scores mean more neutral. All differences are positive: every judge found Grokipedia more biased. Claude was harshest (difference 0.239), Grok mildest (0.041) — softer on its own child, but consistent in direction.
3. The Biases Point in Different Directions
This is the most interesting part. Both encyclopedias are biased, but in different directions:
- Grokipedia: clearly favors economically right-wing figures (β = 0.221, the strongest predictor) and supporters of political pluralism; it is more negative toward supporters of LGBT rights (β = -0.095), women's labor rights (β = -0.050), anti-populism (β = -0.066), and cultural inclusiveness (β = -0.047).
- Wikipedia: leans toward LGBT rights supporters (β = 0.077) and pro-immigration positions (β = 0.036), with no clear favoritism toward the economic right.
4. Inter-Judge Agreement
Full agreement among the four judges was only 23.96%, but adjacent-score agreement (±1) reached 88.24%. Krippendorff's α = 0.403 — "moderate to fair."
Key finding: judges agree on direction (all say Grokipedia is more biased) but differ greatly in strictness. Claude is the strictest judge (only 25% of articles rated neutral), DeepSeek the most lenient (86.9% rated neutral).
Why This Happens
The paper offers no definitive causal explanation, but several clues stand out:
1. Grokipedia Is a "Synthetic Derivative" of Wikipedia
Prior research found Grokipedia is a synthetic derivative of Wikipedia — it draws heavily on Wikipedia but articles are longer (V0.1 averages 15,500 words vs V0.2's 6,500) with worse citation quality. Divergence from Wikipedia is largest on political and controversial topics.
Grokipedia is not an independent knowledge base written from scratch; it's an LLM's reprocessing of Wikipedia — and during that reprocessing, Grok's training-data preferences get injected.
2. LLMs Are Not Blank Slates
Grok's training data itself carries ideological leanings. Research (Buyl et al., "Large language models reflect the ideology of their creators") shows LLMs reflect their creators' ideologies. Training data selection, RLHF annotator preferences, and xAI's alignment strategy all shape output.
Asking a politically-leaning LLM to "neutrally" rewrite an encyclopedia is like asking a partisan editor to "objectively" rewrite a press release — even with effort, subtle wording choices leak the stance.
3. "Neutral" Is Not the Same as "Unbiased"
The paper reveals a deeper issue: neutrality is a contested concept. When Musk says Wikipedia isn't neutral, he means it leans socially liberal; when Grokipedia tries to "correct" that lean, it doesn't become more neutral — it leans the other way.
True neutrality would require balancing across all ideological dimensions, but LLM training has no such explicit constraint. The model just learns to "generate plausible encyclopedic text," and "plausible" is itself a stance-laden concept.
What It Means
A Warning for LLM-Generated Content
Grokipedia isn't an isolated case. Any actor with enough compute can rapidly generate an "encyclopedia." As the paper notes:
> "with enough compute and resources, governments, companies or other actors can rapidly create and distribute extensive knowledge bases, with the risk of such initiatives being used to embed and promote particular ideologies."
This calls for scalable bias audit methods — the paper's framework (multi-LLM judges + expert-coded ideological dimensions + OLS regression) is one reusable approach.
A Warning for LLM Judges
The four judges differed enormously in strictness (Claude: 25% neutral vs DeepSeek: 86.9% neutral). Implications:
1. A single LLM judge's verdict is unreliable. 2. Judge bias and content bias are entangled. 3. Multiple judges from different sources are needed to partially decouple them.
Cold Water on "LLMs Replacing Human Editors"
Human editors' biases stem from editor demographics (predominantly Western, young, white, male) and can be addressed through diversification and transparent processes. LLM biases stem from opaque training data and alignment choices that are hard to debug. Grokipedia shows LLM-generated content isn't "more neutral" — it's *differently biased*: the bias changed direction but didn't disappear.
Limitations
1. Operationalizing neutrality: using LLM judges to assess neutrality risks circularity — LLMs judging LLM bias. 2. V-Party coverage: 9 dimensions can't exhaust all ideological axes. 3. Article selection bias: only government members' biographies were analyzed. 4. Grokipedia V0.2: a beta version; later versions may differ.
Conclusion
Grokipedia is a fable about the illusion of technological neutrality. We like to think machines make more objective decisions — no emotions, no interests, no bias. But machines learn from human-produced data, optimize human-defined objectives, and output "neutrality" as a concept that itself carries ideology.
Grok rated Grokipedia's neutrality lower than its own rating of Wikipedia — not because Grok is self-critical, but because Grokipedia's bias is so pronounced that even the model that produced it could detect it while playing the judge.
That may be the paper's most memorable detail: when bias is large enough, even the bias itself can see it.
---
Paper: https://arxiv.org/abs/2607.15146 HTML version: https://arxiv.org/html/2607.15146v1 Datasets: WhoGov + V-Party