A Dutch government benchmark found no single LLM won every trade-off

A useful answer starts with the question

Which language model should a public body use? A single leaderboard number is a tempting answer. It is also a poor one.

Researchers working with the City of Amsterdam have built a Dutch-language benchmark around a less tidy question: what does the organisation actually need the model to be good at? Their Grip on LLMs framework compares more than 30 models across factuality, honesty, social bias, cost, energy use and transparency about training data.

No model came out best everywhere. Stronger factual answers usually cost more. Bias did not fall neatly as quality or price rose. Some open models exposed more about their training and energy use, while closed API models left important blanks.

That may sound less satisfying than naming a winner. For a government buyer, it is probably more useful.

What Amsterdam measured

The framework grew out of an advisory board, user research and a survey of people using a chatbot for civil servants. The team translated those needs into a public overview aimed at both technical staff and policymakers.

Factuality draws on smaller Dutch versions of MMLU, ARC and TruthfulQA. Honesty is separate: the model is asked things it cannot know or cannot do, then checked for whether it admits the limitation. The study also tests Dutch simplification and summarisation, and looks for bias around age, gender, disability and origin.

For open-weight models, the researchers ran tests on H100 GPUs and used CodeCarbon to estimate electricity per 1,000 prompts. API prices were calculated from token use. Energy for closed API models could not be measured because the underlying infrastructure is not visible.

The live overview also shows licence terms and how much a provider discloses about training data. Those are not model skills, but they can decide whether a public organisation is able to use a system responsibly.

Strong facts did not mean knowing when to stop

The sharpest result is the gap between factuality and honesty. In the paper's tests, GPT-5 scored 0.76 on factuality, the highest in the set, but 0.14 on honesty, the lowest. GPT-4o scored 0.43 on honesty. Mistral Small 24B reached 0.41.

Here, honesty does not mean morality. It means saying, clearly, that a request depends on recent information the model cannot access, specialist knowledge it does not have, or an action it cannot perform.

The authors suggest that alignment for perceived helpfulness might reward confident answers over calibrated uncertainty. That is an explanation, not a tested cause. The benchmark shows the pattern; it does not show why the models produced it.

For public services, the difference matters. A model that gets many benchmark questions right can still be risky if it answers the unknowable ones with the same confidence.

The gaps matter as much as the scores

The paper reports no clear relationship between cost and bias. Paying more did not consistently reduce stereotypical behaviour. The practical lesson is simple: bias cannot be treated as something that disappears when a model gets more capable.

The open-model comparison has a different shape. The authors highlight Mistral Small 24B as a strong open-weight option across their measures, while European initiatives offered clearer training-data disclosure and competitive efficiency at the cost of lower-to-middle quality scores.

Those results belong to this test, at this time. Model versions change. API prices change. A public-benefits assistant, an internal summariser and a tool that drafts letters do not carry the same risks.

The overview is best used to cut a long list down to a few candidates. The next step still has to be a test on the real job, with the people affected by it in the room.

What is confirmed, reported and still open

Confirmed: the five-author paper was submitted on 10 August 2026 and accepted at AIES 2026. The public overview and evaluation code are online, and the repository is published under the European Union Public Licence 1.2.

The research finding: across more than 30 models, no single option led every measured dimension. Factuality and honesty separated, and bias was largely independent of both quality and cost in this evaluation.

The authors' judgement: public bodies should treat the overview as a starting point for model selection, then add task-specific tests and domain expertise. Model Current agrees that the paper supports that cautious use, not a universal ranking.

Still open: how the results transfer to a particular agency, live conversations and newer model releases. Some factuality and summarisation data was translated from English, the bias tests cover only part of Amsterdam's protected grounds, and energy remains unavailable for closed API systems.

A benchmark can make a procurement conversation better. It cannot make the decision on its own.