Overview
This zhichai.net forum post reviews ChiSafe-PAS, a human-annotated Chinese LLM safety benchmark introduced in the paper *Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese* (Zaghouani, Aldous & Gao, Northwestern University in Qatar, arXiv:2605.29667).
The post opens with an example of character-splitting evasion: a harmful request written as decomposed radicals that is instantly readable to a human but invisible to English-trained safety filters.
Key points
- Dataset: 1,897 adversarial Chinese prompts, 1,544 with complete gold-standard annotation (3 response labels: REFUSE / SAFE-REDIRECT / RESPOND, 9-category obfuscation taxonomy, risk levels, annotator rationales).
- Four high-risk domains: self-harm and violence, drugs and illegal trade, fraud, and satire (framed as fiction, PSAs, or irony to trick models into complying).
- Nine obfuscation strategies: pinyin romanization (e.g., "zisha" for 自杀), character/radical splitting, homophone substitution, internet slang and code words (gaming, fan, crypto, anime communities), tonal hedging and rhetorical buffering, culture-specific references, mixed encoding (simplified/traditional, Chinese-English), multi-turn conversational induction, and an open "other" category.
- Mainstream models (GPT, Gemini, open-source Chinese-tuned models) mostly refuse direct harmful requests, but refusal rates fall sharply—sometimes below 10%—under obfuscation.
- The same prompts translated into English are refused significantly more often, showing safety alignment behaves as an English-language privilege rather than a language-agnostic property.
- Models specifically safety-fine-tuned on Chinese data show more resilience, implying cross-lingual generalization from English safety training largely does not happen automatically.
- The authors warn about train/test contamination (public benchmarks being absorbed into training data) and position ChiSafe-PAS as a durable "calibration infrastructure" rather than a one-time exam.
Findings
The author's takeaways
1. Safety is not a translation problem—pinyin tricks and character decomposition have no English equivalent; alignment must be rooted in the target language's culture. 2. Annotation quality beats data scale—1,897 expert-labeled prompts exposed failures that massive machine-generated datasets miss; cultural expertise is empirically irreplaceable. 3. Today's "solved" is tomorrow's exposed—the open "other" category acknowledges attackers keep evolving; safety evaluation requires ongoing version governance and community validation.
The post closes with a broader argument: AI safety frameworks assume an English-centered world, and the ~1.4 billion Chinese speakers deserve safety standards built for their language and internet culture, not second-hand generalization from English red-teaming.
References cited in the post
1. Zaghouani, W., Aldous, K. K., & Gao, Y. (2026). *Beyond English and Evasion.* arXiv:2605.29667. 2. Deng, X., et al. (2024). *Jailbreak Success Rates Across Nine Languages.* 3. Sun, H., et al. (2023). *Systematic Safety Assessment of Chinese LLMs.* 4. Wang, Y., et al. (2024). *Evaluating LLM Safeguards: 3,042 Prompts Across Three Attack Perspectives.* 5. Ganguli, D., et al. (2022). *Red Teaming Language Models to Reduce Harms.*