Eight legal RAG systems still hallucinated with the law in front of them

Retrieval helped. It did not settle the answer

Putting a legal source in front of a language model sounds like a sensible way to keep it grounded. It is. It is not a guarantee.

A new study examined eight legal systems that retrieve relevant law before writing an answer. Every setup still produced unsupported or contradicted claims. In the strongest result, 8.3% of answers about a French civil-law corpus contained at least one such claim. In the weakest, the rate reached 48.3% on GDPR questions.

The errors were often small. Most affected answers contained one bad claim rather than falling apart from start to finish. That may be the more awkward failure: a mostly useful answer can make the unsupported sentence look trustworthy too.

The study does not show that retrieval is useless. It shows why a citation or a retrieved passage should not be treated as proof that every sentence follows from it.

Eight systems, two bodies of law

The researchers worked from ClaimRAG-LAW, a benchmark with 317 expert-validated question-and-answer pairs. It covers the European Union's GDPR in English and a national civil-law corpus in French.

They combined two retrieval methods, BM25 and E5, with four generators: Llama 3.1 8B Instruct, Mixtral 8x7B, GPT-4 and GPT-5. That produced eight systems with the same basic job but different ways of finding and writing the answer.

Each response was broken into factual claims and checked against the reference material. The paper reports both claim-level density and answer-level frequency. The first asks how much of an answer was unsupported. The second asks whether an answer contained at least one unsupported or contradicted claim.

On that answer-level measure, BM25 with GPT-5 recorded 20.1% on the GDPR set and 8.3% on the French civil-law set. E5 with Llama 3.1 8B reached 48.3% on GDPR. BM25 with the same generator reached 45.8% on the French corpus.

The isolated sentence is the hard one to notice

The stronger systems did not simply write less. BM25 with GPT-5 produced the most claims per GDPR answer on average and still recorded the lowest claim-level hallucination density there, at 1.5%.

That is encouraging, but the answer-level number tells a different story. One unsupported claim is enough to make an answer unreliable for a person who does not know which sentence needs checking.

Severe cases were rarer. Even so, the paper found isolated answers from GPT-5-based systems containing eight or nine unsupported claims. A low average can therefore sit beside a small number of serious misses.

This distinction matters in legal work. The user may not need a perfect essay. They do need to know which proposition came from the source and which one was filled in by the model.

A bad assumption can pull the whole answer off course

False-premise questions were a recurring problem. These questions contain an incorrect assumption that a useful system should reject before doing anything else.

One example asked how the GDPR requires controllers and processors to submit assessments and safeguards to a public registry. The GDPR contains no such requirement. Several systems accepted the premise and built an answer around it.

The paper also found lower hallucination rates for questions associated with legal experts than for some questions written for civil officers or citizens. The authors argue that wording, intent and domain knowledge all affect what the system produces.

There is an important caution here. ClaimRAG-LAW contained only 13 false-premise questions and three jurisdiction-or-time-specific questions. The direction is useful. The exact rates for those small categories are not yet stable evidence.

What is confirmed, measured and still open

Confirmed: the three-author preprint was submitted on 14 August 2026. It analyses eight retriever-generator combinations across 317 expert-validated benchmark questions and tests the broad pattern again on 142 independently drafted legal questions.

Measured in this setup: every system produced unsupported or contradicted claims. Answer-level rates ranged from 8.3% to 48.3% across the two corpora and eight configurations. Most affected answers contained an isolated bad claim rather than many of them.

The researchers' interpretation: legal RAG should be evaluated claim by claim and across question types, not only with one aggregate score. Retrieval quality, generator choice and the user's wording all matter.

Important limits: the study re-analysed benchmark outputs rather than rerunning the systems. It covered two legal sources, two retrievers and four generators. It did not vary prompts, chunking, retrieval settings or newer model versions.

Still open: whether the same ranking holds in other jurisdictions and live legal products, and which interface makes an isolated unsupported claim visible before someone relies on it. Retrieval narrows the room for invention. It does not close it.