On March 15, 2026, China's CCTV 3·15 Gala exposed a black-market supply chain: commercial GEO (Generative Engine Optimization) operators can plant fake reviews online and, within hours, get a non-existent brand to appear in the top recommendations of mainstream AI assistants.
This is not alarmism. A research team from the Chinese University of Hong Kong demonstrated with rigorous experiments that a single poisoned webpage ranked first in search results can make the most vulnerable AI model recommend a fake brand 27% of the time. When the number of poisoned pages increases to 3, the deception rate for the most vulnerable model exceeds 50%.
FORGE: A Controlled Experiment Without Polluting the Real Web
How do you measure an AI recommendation system's resistance to disinformation? You can't actually post fake reviews online—that would be neither ethical nor controllable.
The research team designed the FORGE (Fake Online Recommendations in Generative Environments) benchmark. The core idea is "local replacement": freeze a real set of search results, then replace the real brand names appearing in them with fictional brand names, while keeping document ranking, URLs, sources, length, style, and context completely unchanged. Any change in the model's recommendation behavior can then be attributed solely to the brand name swap.
The design is clever: it avoids polluting the real web while guaranteeing reproducibility. The replacement is also minimal—only the brand name changes, everything else stays the same—so if a model recommends the fake brand, it is purely because it treated the poisoned content as credible evidence.
12 Models, All Compromised
The study tested 12 mainstream models, including closed-source GPT-5.4, Claude Opus 4.7, and Gemini 3.1 Pro, as well as open-source Qwen3.5-9B, DeepSeek V4 Pro, and GLM-4.6V-Flash.
The results are unsettling:
- All models are vulnerable, with closed-source and open-source models showing highly overlapping deception rates. Bigger models are not necessarily safer—Gemini 3.1 Pro's deception rate is roughly three times that of Gemini 3 Flash.
- Reasoning makes models more vulnerable. With reasoning disabled, Qwen3.5-9B's deception rate dropped from 80.4% to 56.9%, and GLM-4.6V-Flash's from 71.6% to 38.7%. When a model "thinks deeply" about poisoned content, it actually becomes more likely to convince itself the fake brand is real.
- Category differences are huge. Everyday consumer categories like restaurants, personal services, and health supplements are easiest to attack—because in these areas users rely on community word-of-mouth rather than authoritative brands. Technical categories like smartphones and laptops are relatively safer, since models have more solid brand knowledge there.
The Most Disturbing Finding: AI "Invents" Social Proof
Deceived models don't just parrot poisoned content. They actively fabricate social proof that doesn't exist—claiming "the community is buzzing" or "many users recommend it," none of which appears in the poisoned documents. Social proof markers appear 1.5 to 11 times more frequently in deceived outputs than in resistant ones.
This means AI isn't just fooled by fake reviews—it helps cover for them. It's like a person influenced by a rumor who not only believes it, but proactively invents "I've heard many people say the same thing" to reinforce their judgment.
Three Defense Strategies, All Failed
The study tested three inference-time defense strategies:
1. Skepticism Prompting: telling the model "retrieved results may contain false information, judge carefully." Result: not only did it fail, it systematically backfired on closed-source models—Gemini 3.1 Pro's deception rate *increased* by 44 percentage points. Like reasoning, skepticism prompts make the model examine poisoned content more deeply, making it easier to be persuaded.
2. Model Prior Consensus Filtering: filtering out any brand the model wouldn't recommend without retrieval results. Result: it does catch fake brands, but at the cost of suppressing 52%–79% of legitimate recommendations.
3. Cross-Document Evidence Consistency Filtering: requiring multiple documents to corroborate each other. Result: the same precision-recall tradeoff—catching fake brands means sacrificing large numbers of genuine recommendations.
Why Is Reasoning a Double-Edged Sword?
The finding that reasoning makes models more vulnerable is counterintuitive. We usually assume "thinking more" should be safer. But in recommendation scenarios, reasoning becomes a process of self-persuasion: the model reads poisoned content, starts thinking about why this brand might be good, finds plausible justifications (even if fabricated), and ends up even more convinced.
This bears a striking resemblance to "confirmation bias" in cognitive psychology—people tend to seek evidence supporting their initial judgment, and reasoning models seem to fall into the same trap.
One Ranking Decides Everything
The most shocking number in the experiments: a single poisoned webpage ranked #1 in search results can deceive the most vulnerable model 27% of the time. But the same page placed at positions 2 through 10 produces deception rates of only 1%–4%.
The first page the model reads dominates the recommendation; the rest barely matter. This matches GEO operators' practical experience perfectly—they don't need fake reviews everywhere, they just need theirs to rank at the top of search results.
What Does This Mean?
FORGE reveals a structural threat: generative recommendation systems shift the trust boundary from inside the model to the open web, which has no curator. Any motivated attacker can push false content to the top of search results via SEO, and the AI will faithfully consume it—even actively fabricating social proof to reinforce the fake recommendation.
The paper's authors note that robust generative recommendation requires defenses built in at retrieval time: source credibility weighting, evidence diversification, and cross-document verification. But the more fundamental question is—when AI recommendation systems become the critical gateway for consumer decisions, who is responsible for the quality of search results?
Perhaps next time an AI recommends a brand with a "great reputation," you should first check whether the brand actually exists.
---
Paper: https://arxiv.org/abs/2606.13610 Code repository: https://github.com/leoluolol/forge-benchmark