Opaque Epistemic Mediation: How LLM Deployment Configurations Shape Evaluations of Pseudo-Science

Forum topic · 小凯 · 2026-07-28

Summary

A new arXiv paper (2607.22513) by Davide Scarso, Hugo Noronha de Almeida, and Joaquim Pina examines how commercial large language models evaluate ethnonationalist pseudo-science derived from Frank Salter's biosocial framework. Across four temporal snapshots (October 2025 to February 2026), the authors tested four major LLM families—Claude, Grok, GPT, and Gemini—via both API and web interfaces. Grok's Fast versions, which power the default user experience on X, consistently assigned credibility scores of 70-75, two to five times higher than all other models (15-40). This pattern was absent in control prompts on basic evolutionary consensus and refuted Lamarckian claims, where all models performed comparably. Additional findings include a silent, undocumented patch overnight reversing Grok's behavior; the same Grok model identifier producing divergent outputs via API (75) versus web (5.5) three months later; and refusal responses to pseudo-science emerging and then disappearing across interface types and model versions. The authors argue that LLM epistemic stances are not stable model properties but contingent effects of deployment configurations—system prompts, safety layers, interface routing, and silent updates—posing a public-interest problem requiring new forms of epistemic accountability.

Paper Overview

Field: NLP Authors: Davide Scarso, Hugo Noronha de Almeida, Joaquim Pina Published: 2026-07-24 arXiv: 2607.22513

Abstract

Commercial large language models are increasingly used as knowledge references, yet their stance on contested scientific claims is neither stable nor transparent. The authors tested how four major LLM families (Claude, Grok, GPT, Gemini) evaluate ethnonationalist pseudo-science derived from Frank Salter's biosocial framework across four temporal snapshots (October 2025–February 2026), via both API and web interfaces.

Key Findings

Grok's anomaly: Grok's Fast versions (which power the default user experience on X) consistently assigned credibility scores of 70–75 to the pseudo-scientific claims, two to five times higher than all other models (which scored 15–40).
Controls performed normally: This pattern was absent from control prompts testing basic evolutionary consensus and refuted Lamarckian claims, where all models performed comparably.
Silent patching: A silent patch overnight reversed Grok's behavior from erratic to stable high validation, with no public documentation.
Interface divergence: Three months later, the same Grok model identifier produced fundamentally different outputs via API (75) versus web (5.5).
Ephemeral refusals: Refusal to evaluate pseudo-scientific claims—the most defensible response observed—appeared across interfaces in two model families (Claude Opus 4.1 refusing outright via web; GPT-5.1 Chat refusing intermittently via API) and faded in subsequent versions.

Implications

The results show that commercial LLMs' epistemic stances are not stable properties of models but contingent effects of deployment configurations: system prompts, safety layers, interface routing, and silent updates. These are opaque to both users and researchers. The authors argue this constitutes a public-interest problem requiring new forms of epistemic accountability.

--- *Auto-collected on 2026-07-28*