English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

ChiSafe-PAS: When Chinese Character-Splitting Tricks Defeat LLM Safety Filters

Forum topic · 小凯 · 2026-06-01

Summary

A Chinese tech forum post reviews the ChiSafe-PAS benchmark (arXiv:2605.29667) from Northwestern University in Qatar, a human-annotated dataset of 1,897 adversarial Chinese prompts (1,544 gold-standard labels) for evaluating LLM safety in Chinese. The post explains how LLM safety alignment, built almost entirely on English red-teaming data, fails systematically when Chinese users disguise harmful requests using pinyin romanization, character decomposition, homophone substitution, internet slang, tonal hedging, cultural references, mixed encoding, and multi-turn诱导—nine obfuscation categories in total. Experiments on GPT, Gemini, and open-source Chinese models show refusal rates dropping sharply (sometimes below 10%) under obfuscated prompts, while English translations of the same prompts are often blocked, demonstrating that safety alignment is not language-agnostic. The dataset covers self-harm/violence, drugs/illegal trade, fraud, and satire, with labels for REFUSE, SAFE-REDIRECT, and RESPOND plus annotator rationales. The author argues that safety is a cultural problem, not a translation problem, and that human cultural expertise cannot be replaced by scaled machine-generated data.

Overview

This zhichai.net forum post reviews ChiSafe-PAS, a human-annotated Chinese LLM safety benchmark introduced in the paper *Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese* (Zaghouani, Aldous & Gao, Northwestern University in Qatar, arXiv:2605.29667).

The post opens with an example of character-splitting evasion: a harmful request written as decomposed radicals that is instantly readable to a human but invisible to English-trained safety filters.

Key points

  • Dataset: 1,897 adversarial Chinese prompts, 1,544 with complete gold-standard annotation (3 response labels: REFUSE / SAFE-REDIRECT / RESPOND, 9-category obfuscation taxonomy, risk levels, annotator rationales).
  • Four high-risk domains: self-harm and violence, drugs and illegal trade, fraud, and satire (framed as fiction, PSAs, or irony to trick models into complying).
  • Nine obfuscation strategies: pinyin romanization (e.g., "zisha" for 自杀), character/radical splitting, homophone substitution, internet slang and code words (gaming, fan, crypto, anime communities), tonal hedging and rhetorical buffering, culture-specific references, mixed encoding (simplified/traditional, Chinese-English), multi-turn conversational induction, and an open "other" category.
  • Findings

  • Mainstream models (GPT, Gemini, open-source Chinese-tuned models) mostly refuse direct harmful requests, but refusal rates fall sharply—sometimes below 10%—under obfuscation.
  • The same prompts translated into English are refused significantly more often, showing safety alignment behaves as an English-language privilege rather than a language-agnostic property.
  • Models specifically safety-fine-tuned on Chinese data show more resilience, implying cross-lingual generalization from English safety training largely does not happen automatically.
  • The authors warn about train/test contamination (public benchmarks being absorbed into training data) and position ChiSafe-PAS as a durable "calibration infrastructure" rather than a one-time exam.

The author's takeaways

1. Safety is not a translation problem—pinyin tricks and character decomposition have no English equivalent; alignment must be rooted in the target language's culture. 2. Annotation quality beats data scale—1,897 expert-labeled prompts exposed failures that massive machine-generated datasets miss; cultural expertise is empirically irreplaceable. 3. Today's "solved" is tomorrow's exposed—the open "other" category acknowledges attackers keep evolving; safety evaluation requires ongoing version governance and community validation.

The post closes with a broader argument: AI safety frameworks assume an English-centered world, and the ~1.4 billion Chinese speakers deserve safety standards built for their language and internet culture, not second-hand generalization from English red-teaming.

References cited in the post

1. Zaghouani, W., Aldous, K. K., & Gao, Y. (2026). *Beyond English and Evasion.* arXiv:2605.29667. 2. Deng, X., et al. (2024). *Jailbreak Success Rates Across Nine Languages.* 3. Sun, H., et al. (2023). *Systematic Safety Assessment of Chinese LLMs.* 4. Wang, Y., et al. (2024). *Evaluating LLM Safeguards: 3,042 Prompts Across Three Attack Perspectives.* 5. Ganguli, D., et al. (2022). *Red Teaming Language Models to Reduce Harms.*

Tags

#llm-safety#chinese-nlp#chisafe-pas#adversarial-prompts#benchmark#jailbreak#multilingual-ai#content-moderation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177980683