From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Field: NLP Authors: Laurens Samson, Iva Gornishka, Gossa Lô Published: 2026-08-11 arXiv: 2508.03803
Summary
Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. The authors present the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation.
Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, the authors identify six evaluation dimensions:
- Factuality
- Honesty
- Social bias
- Energy consumption
- Cost
- Training data transparency
- No single model excels on all dimensions. Trade-offs are unavoidable: higher quality consistently comes with greater environmental impact and financial cost, while social bias is largely independent of both.
- Factuality and honesty are distinct. Whether a model answers correctly (factuality) and whether it admits it does not know (honesty) are driven by different model attributes; high factuality does not imply high honesty.
- Actionable outputs for non-technical audiences. The authors release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection, from engineers to policymakers.
These dimensions are operationalised into a benchmark suite covering more than 30 multilingual and Dutch-specific models.
Key Findings
Significance
The work addresses a gap in LLM evaluation by combining public-administration values with non-English (Dutch) linguistic requirements, offering a replicable methodology for other governmental and multilingual contexts.
--- *Auto-collected on 2026-08-12*