Grip on LLMs: Benchmarking Large Language Models for Dutch Government Use

Grip on LLMs: Benchmarking Large Language Models for Dutch Government Use

Research area: NLP Authors: Laurens Samson, Iva Gornishka, Gossa Lô arXiv: 2508.03803

Summary

Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. The authors present the "Grip on LLMs" framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation.

Methodology

Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, the team identified six evaluation dimensions:

• Factuality – whether the model answers correctly
• Honesty – whether the model acknowledges not knowing
• Social bias
• Energy consumption
• Cost
• Training data transparency

These dimensions were operationalised into a benchmark suite covering more than 30 multilingual and Dutch-specific models.

Key Findings

• No single model excels across all dimensions; trade-offs are unavoidable.
• Higher quality consistently comes with greater environmental and financial cost, while bias levels are largely independent of these factors.
• Factuality and honesty are driven by different model properties: high factuality does not imply high honesty.

Deliverable

To make the findings actionable for non-technical audiences, the authors release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection—from engineers to policymakers.

--- *Auto-collected on 2026-08-12*

Key Points

• New evaluation framework tailored to Dutch government LLM deployment
• Six dimensions: factuality, honesty, bias, energy, cost, transparency
• Covers 30+ multilingual and Dutch-specific models
• Trade-offs between quality, environmental impact, cost, and bias
• Factuality and honesty are independent model properties
• Public tool released for engineers and policymakers