When AI Writes Poems Better Than Humans: Detecting AI-Generated Chinese Poetry by Making It "Look" at a Picture
"The white sun sets behind the mountains; the Yellow River flows into the sea." For a thousand years, Chinese poets have followed the principle of "expressing emotion through scenery" — first there are the landscapes before the eyes, then the feelings in the heart. Poetry comes from the image; the image is poetry's root.
But today's large language models write poems that seem more convincingly "poetic" than much human writing. They have never seen mountains or rivers, never experienced joy or sorrow, yet they can generate well-formed verses with beautiful imagery. Traditional detection methods — word frequency analysis, sentence pattern checks, perplexity — are losing accuracy against increasingly capable models. Most detectors can't even reach 70%.
If text detection is failing, can we return to the source of poetry — the "image" itself?
A research team from Renmin University of China and Tencent proposed a clever idea: have a multimodal large language model (MLLM) "look" at the picture corresponding to a poem, then judge whether the poem was written by a human or by AI. The framework is called IMAGINE (Image-seMantic guIded detectioN of ai-gEnerated poetry) — and imagination, it turns out, is exactly what AI finds hardest to fake.
---
Why Isn't Text Detection Enough?
Traditional AI text detection relies on two types of clues:
1. Statistical features: AI-generated text tends to have lower perplexity, more uniform word frequency distributions, and more regular repetition patterns. 2. Semantic features: AI tends to produce "safe" semantic combinations, avoiding overly jumpy or contradictory imagery.
But poetry is precisely the exception to these rules. Good poetry is *supposed* to break conventions — in "red apricot blossoms bustle on the branch," a single word turns the visual into the auditory. This kind of synesthesia looks like an anomaly in statistical terms. Meanwhile, AI poems that deliberately chase the "feel" of poetry can come across as overly polished, overly "correct."
The result: text detectors fail especially badly on poetry. A "too polished" human poem may be misjudged as AI; a "too jumpy" AI poem may be misjudged as human. Both directions go wrong.
---
IMAGINE: Making AI "See" the Picture Behind the Poem
IMAGINE's core insight comes from the most fundamental creative principle of Chinese poetry: "There is painting in poetry, and poetry in painting."
Human poets write from visual experience (what they saw) transformed into language (what they wrote). This "vision → language" conversion leaves deep traces — the imagery combinations, spatial relations, and color pairings in a poem are all constrained by the original visual scene.
AI poetry, by contrast, is generated directly in language space — it never "saw" anything; it just samples from a probability distribution. It may produce a beautiful line like "bright moon shines through the pines, clear spring flows over the stones," but it cannot answer: from which direction does the moonlight come? On which side of the spring are the pines?
IMAGINE exploits exactly this gap. The pipeline:
1. Acquire the image: pair each poem with a corresponding picture (from a poem-image dataset) 2. Multimodal input: feed the poem text plus the image into an MLLM 3. Semantic consistency judgment: the MLLM evaluates whether the visual scene matches the imagery described in the poem 4. Combined verdict: merge text features with image-text consistency for a final decision
Key assumption: human poetry's imagery is constrained by real visual scenes, yielding high image-text consistency; AI poetry's imagery is a product of language probabilities, with lower consistency against real visual scenes.
---
The Dataset: 800 Human Poems + 3,200 AI Poems
To validate IMAGINE, the team built a dedicated poetry detection dataset:
- Human poems: 800, drawn from classical and modern Chinese poetry
- AI poems: 3,200, generated by four mainstream LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, GLM-4)
- Images: 800, visual scenes corresponding to the human poems
---
Results: Adding Images Boosts Detection Accuracy Sharply
| Method | Macro-F1 | |--------|----------| | Best text-only detection | ~70% | | Gemini + IMAGINE | 85.65% | | Improvement | +15.37% |
IMAGINE lifts detection accuracy from ~70% to 85.65% — a gain of more than 15 percentage points.
Finer-grained analysis revealed several interesting findings:
1. Different models have different "tells": Poems generated by GPT-4o are the hardest to detect, closest to human in semantic coherence; some models tend to slip up on rare imagery combinations, producing visually incoherent pairings.
2. The image acts as a corrector: Text-only detection tends to misjudge "too well-written" human poems as AI. With images added, the MLLM can verify whether the "good" imagery actually comes from real visual experience, correcting such misjudgments.
3. MLLM choice matters: Gemini performed best within IMAGINE, likely due to its stronger multimodal understanding and more accurate image-text consistency evaluation.
---
A Deeper Question: Are There Limits to AI's "Imagination"?
IMAGINE's success hints at a deeper question: does AI's "imagination" have a boundary?
A human poet wrote "a lone smoke column rises straight over the vast desert; the setting sun hangs round over the long river" because Wang Wei actually stood on that northwestern desert and saw the scene. Every image — desert, lone smoke, long river, setting sun — belongs to one unified visual scene, self-consistent in space, time, and light.
AI can produce the same words, but it cannot guarantee these images are visually self-consistent. What should "lone smoke" and "setting sun" look like in the same picture? Is the smoke's direction consistent with the sun's position? Only something that has "truly seen" handles such details naturally.
At its core, IMAGINE tests whether AI has "visual common sense" — not linguistic common sense, but visual common sense of the physical world.
This also explains why pure text detection is getting harder: AI's language ability is improving rapidly, but its "visual grounding" remains weak. As long as we return to the origin that "poetry comes from painting," we can find AI poetry's weak spot.
---
An Honest Assessment
The paper's approach is genuinely clever — instead of grinding harder on the text-detection track, it leverages the essential feature of Chinese poetry, "expressing emotion through scenery," to open an entirely new detection path.
Still, several points deserve scrutiny:
1. Dependence on paired images: IMAGINE requires an image for each poem, which may be inconvenient in practice; poor-quality or mismatched images could hurt performance. 2. Cross-language generalization: It has only been validated on Chinese poetry; applicability to English or other languages is unclear. 3. Adversarial attacks: If AI models learn that the detector looks at images, they could "reverse engineer" the approach — first imagine a visual scene, then generate a poem grounded in it, improving image-text consistency. 4. The standard for "good poetry": Detecting AI poetry and judging poetic quality are different things. A poem judged as "human-written" is not necessarily a good poem.
Regardless, IMAGINE reminds us that what AI struggles hardest to fake is not language, but experience. When we use the thousand-year-old wisdom of "painting within poetry" to detect AI, we are really saying: true creativity comes from genuine perception of the world, not from precise computation over probabilities.
---
📎 Paper: Seeing the Poem: AI-Generated Chinese Poetry Detection via Image-Semantic Guided Multi-Modal LLM
📎 HTML version: arxiv.org/html/2605.22654v1