Overview
- Field: NLP
- Authors: Laurens Samson, Iva Gornishka, Gossa Lô
- Published: 2026-08-11
- arXiv: 2508.03803
- Methodology: An advisory board process, user research, and a survey of civil-servant chatbot users were used to identify evaluation dimensions grounded in public-administration values.
- Six evaluation dimensions: factuality, honesty, social bias, energy consumption, cost, and training data transparency.
- Benchmark scope: The dimensions are operationalized into a benchmark suite covering more than 30 multilingual and Dutch-specific models.
- Trade-offs: No single model excels across all dimensions. Higher quality consistently comes with greater environmental impact and financial cost, while social bias is largely uncorrelated with both.
- Factuality vs. honesty: These are driven by different model attributes — a model that answers correctly (high factuality) does not necessarily admit when it does not know (high honesty).
- Practical output: A publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in government LLM selection, from engineers to policymakers.
Summary
Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. The authors present the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation.
Key points
Significance
The framework addresses two gaps simultaneously: aligning LLM evaluation with public-sector values and accounting for non-English (Dutch) linguistic requirements. It offers a replicable template for other governments evaluating LLMs in their own languages.
*Paper*: arXiv:2508.03803