On March 15, 2026, China's CCTV 3·15 Gala exposed a black-market industry: commercial GEO (Generative Engine Optimization) operators can make a non-existent brand appear in mainstream AI assistants' hot recommendations within hours by planting fake reviews across the web.
This is no exaggeration. A research team from The Chinese University of Hong Kong demonstrated with rigorous experiments that a single poisoned webpage ranked first in search results can make the most vulnerable AI model recommend a fake brand 27% of the time. With three poisoned pages, the deception rate for the most vulnerable models exceeds 50%.
FORGE: A Controlled Benchmark Without Polluting the Real Web
How do you measure an AI recommendation system's resistance to misinformation? You can't actually post fake reviews online—that's neither ethical nor controllable.
The team designed FORGE (Fake Online Recommendations in Generative Environments). The core idea is "local replacement": freeze a real search result set, then replace the real brand names appearing in it with fictional brand names, while keeping document rankings, URLs, sources, length, style, and context completely unchanged. Any change in the model's recommendation behavior can then only be attributed to the brand name substitution.
The design is clever: it avoids polluting the real web while guaranteeing reproducibility. And because the substitution is minimal—only brand names change—if a model recommends a fake brand, it is purely because it treated the poisoned content as credible evidence.
12 Models, All Compromised
The study tested 12 mainstream models, including closed-source GPT-5.4, Claude Opus 4.7, and Gemini 3.1 Pro, plus open-source Qwen3.5-9B, DeepSeek V4 Pro, GLM-4.6V-Flash, and others.
The results are unsettling:
- All models are vulnerable, with highly overlapping deception rates between closed- and open-source models. Bigger models are not necessarily safer—Gemini 3.1 Pro's deception rate is roughly three times that of Gemini 3 Flash.
- Reasoning makes models more vulnerable. With reasoning disabled, Qwen3.5-9B's deception rate dropped from 80.4% to 56.9%, and GLM-4.6V-Flash's from 71.6% to 38.7%. When a model "thinks hard" about poisoned content, it becomes more likely to convince itself the fake brand is real.
- Huge category differences. Everyday consumer categories—restaurants, personal services, health supplements—are easiest to attack, because users rely on community word-of-mouth rather than authoritative brands. Technical categories like smartphones and laptops are relatively safe, since models have firmer brand knowledge there.
The Scariest Finding: AI "Invents" Social Proof
Deceived models don't just copy poisoned content. They actively fabricate social proof that doesn't exist—claiming "lively community discussions" or "many user recommendations" that appear nowhere in the poisoned documents. Social-proof markers appear 1.5x to 11x more frequently in deceived outputs than in resisting outputs.
This means AI isn't merely fooled by fake reviews—it helps cover for them. Like a person influenced by a rumor who not only believes it but invents "I've heard many people say the same thing" to reinforce their judgment.
Three Defense Strategies, All Failed
The study tested three inference-time defenses:
1. Skepticism Prompting: telling the model "retrieved results may contain misinformation, judge carefully." Result: not only ineffective, but systematically counterproductive on closed-source models—Gemini 3.1 Pro's deception rate actually *increased* by 44 percentage points. Like reasoning, skepticism prompts make the model examine poisoned content more deeply, making it easier to be persuaded.
2. Prior-consensus filtering: if the model wouldn't recommend a brand without retrieval results, filter it out. Result: it does catch fake brands, but at the cost of suppressing 52%–79% of legitimate recommendations.
3. Cross-document evidence consistency filtering: requiring multiple documents to corroborate each other. Result: the same precision-recall tradeoff—catching fake brands means sacrificing large numbers of genuine recommendations.
Why Reasoning Is a Double-Edged Sword
Reasoning making models more vulnerable is counterintuitive. Normally we assume "thinking more" is safer. But in recommendation scenarios, reasoning becomes a process of self-persuasion: the model reads poisoned content, starts considering why this brand might be good, finds plausible justifications (even fabricated ones), and ends up more convinced.
This strikingly mirrors "confirmation bias" in cognitive psychology—people tend to seek evidence supporting their initial judgment, and reasoning models appear to fall into the same trap.
One Ranking Decides Everything
The most shocking number in the experiments: a single poisoned page ranked #1 in search results deceives the most vulnerable models 27% of the time. But the same page at positions 2 through 10 yields only a 1%–4% deception rate.
The first page a model reads dominates the recommendation; the rest barely matter. This matches GEO operators' practical experience perfectly—they don't need floods of fake reviews, just their fake reviews ranked at the top of search results.
What Does This Mean?
FORGE reveals a structural threat: generative recommendation systems shift the trust boundary from inside the model to the open web, and the open web has no curator. Any motivated attacker can push misinformation to the top of search results via SEO, and AI will faithfully consume it—even fabricating social proof to reinforce the fake recommendation.
The paper's authors note that robust generative recommendation requires defenses at retrieval time: source credibility weighting, evidence diversification, and cross-document verification. But the more fundamental question is: when AI recommendation systems become a critical gateway for consumer decisions, who is accountable for the quality of search results?
Perhaps the next time an AI recommends a brand with "great word-of-mouth," you should first search whether the brand actually exists.
---
Paper: https://arxiv.org/abs/2606.13610 Code repository: https://github.com/leoluolol/forge-benchmark