ML-Bench & Guard: Policy-Grounded Multilingual Safety Benchmark and Guardrails for LLMs

Forum topic · 小凯 · 2026-05-04

Summary

ML-Bench & Guard introduces a policy-grounded approach to evaluating and enforcing large language model (LLM) safety across languages and regions. Unlike existing safety benchmarks that are mostly English-centric, rely on generic risk taxonomies, and extend to other languages via machine translation, ML-Bench builds test cases directly from real laws, regulations, and platform policies. It covers 14 languages, with each language independently collected and annotated natively rather than translated, preserving culture-specific nuance. The benchmark is designed to be dynamically updatable as policies evolve. Its companion model, ML-Guard, is a guardrail trained on ML-Bench that answers region-specific compliance questions (e.g., whether content violates a given policy) instead of producing simple binary harm labels. The paper argues this addresses three gaps in multilingual AI safety: the language gap (English-only benchmarks and poor translation quality), the regulatory gap (generic taxonomies that miss region-specific legal requirements), and the cultural gap (definitions of harm vary across cultures). Policy grounding also improves compliance evidence, cultural adaptability, and explainability, since refusals can cite specific policy clauses. The core takeaway: LLM safety must be grounded in real-world laws and cultural norms rather than abstract, one-size-fits-all risk categories.

Paper: ML-Bench&Guard: Policy-Grounded Multilingual Safety Benchmark and Guardrail for Large Language Models Authors: Yunhan Zhao, Zhaorun Chen, Xingjun Ma, Yu-Gang Jiang, Bo Li arXiv: 2605.00689 (2026-04-30)

The Problem with One-Size-Fits-All Safety Standards

A globally deployed LLM must respect US law in the US, GDPR in the EU, local content regulations in China, and cultural taboos in the Middle East. Yet most existing safety benchmarks:

Are primarily English-based
Use generic risk taxonomies (e.g., "violence", "hate")
Extend to other languages via machine translation
Ignore region-specific regulations and cultural differences

The result: a model judged "safe" in the US may be "unsafe" elsewhere.

Three Gaps in Multilingual AI Safety

1. Language gap — English-dominant benchmarks, inconsistent machine translation, untranslatable culture-specific concepts. 2. Regulatory gap — different regions have different laws; generic taxonomies miss region-specific requirements (e.g., religious taboos in some countries). 3. Cultural gap — what counts as "safe" varies by culture; translation cannot capture cultural nuance.

Existing guardrail models are also limited: constrained to predefined risk categories, unable to adapt to new regional requirements, and lacking policy-grounded capabilities.

ML-Bench: A Policy-Grounded, Multilingual Benchmark

Key innovations:

Built directly from real policies — test cases derive from actual laws, regulations, and platform policies rather than handcrafted scenarios, covering 14 languages.
Natively multilingual — each language is independently collected and annotated, not translated from English, preserving cultural specificity and contextual accuracy.
Dynamically extensible — when policies change or new regulations emerge, the benchmark can be updated. It is a "living" benchmark, not a static one.

ML-Guard: The Companion Guardrail

Trained on ML-Bench, ML-Guard understands and enforces region-specific policies. Instead of a simple yes/no classification, it answers: "According to policy X, is this content compliant?"

It is like giving an AI an international legal team — knowing not just "what is bad," but "what cannot be done under local law."

Why Policy Grounding Matters

Generic taxonomies fail because "hate speech" means different things in the US, Germany, and India; political, religious, gender, and historical topics carry vastly different sensitivities across countries. Policy grounding offers:

1. Compliance — direct alignment with real regulations; enterprises can demonstrate compliance and reduce legal risk. 2. Cultural adaptability — respect for local norms instead of a single cultural default; higher user acceptance. 3. Explainability — refusals can cite specific clauses ("this violates Article Y of Regulation X") rather than a vague "harmful content detected."

Key Takeaway

The paper's philosophy echoes Feynman's principle: if your understanding does not match reality, your understanding is wrong. Abstract safety benchmarks test theoretical correctness; policy-grounded safety delivers reliability in practice.

If you build or evaluate multilingual AI systems, ask:

Does my safety benchmark reflect real regulations, or handcrafted abstract categories?
Is my multilingual support machine-translated or natively constructed?
Can my guardrail adapt to different regional requirements?
Does my evaluation account for cultural specificity?

The future of AI safety is not "one standard for the world" but flexible, policy-grounded adaptation. Multilingual AI safety begins with respect for cultural diversity.

---

*Source: zhichai.net forum post (Chinese tech forum). Paper link: arXiv 2605.00689.*