The average looked quiet. Some records did not stay quiet
A language model can look unremarkable on an average privacy test and still reproduce a specific piece of its training data.
That is the useful warning in a new preprint from Victor Maricato at Karolinska Institutet. The study tested open language models using only generated samples, the sort of black-box access available through an ordinary inference endpoint.
Its clearest experiment used 500 documents from the Pile that contained a real email address or telephone number. The researcher placed the prompt immediately before the identifier, generated 32 continuations and checked whether the model completed the exact string.
On Pythia-6.9B, that happened for 83 documents. The reported rate was 16.6%, with a 95% interval from 13.6% to 20.1%. Among the 300 email documents, the model reproduced 64 addresses, or 21.3%.
No addresses, phone numbers or raw completions were released. The paper reports masked totals and per-document booleans instead.
A control separated a record from a common string
A model may emit a common address in many contexts. That would not show that one particular training record had resurfaced.
So the study added a mismatched-prefix control. Each identifier was also tested behind text taken from another document. A case counted as a document-specific leak only when the identifier appeared with its own prefix and did not appear with the mismatched one.
This is a fairly capable attacker. They already hold the text just before the missing identifier. The experiment asks whether the model can fill in the hidden part, not whether a stranger can discover an arbitrary person's data from nothing.
That distinction keeps the result concrete. It models partial-record access, which can matter in an audit or a targeted attack. It does not establish a universal extraction rate for casual prompts.
The paper's result is also about exact reproduction. It is different from asking whether an example was probably in the training set, the task usually measured by membership inference.
Why one AUC number missed the exposed tail
Membership-inference work is often compressed into an area-under-the-curve score. The score describes how well a method separates training examples from non-members across a whole collection.
Here, that average was a poor guide. On the WikiMIA benchmark, a model-free bag-of-words baseline reached an AUC of 0.97 simply from surface differences between the two text groups. Sampling from the model added no useful signal.
On the more carefully matched MIMIR split, none of the tested sampling measurements significantly improved on a blind baseline. Yet a small set of individual prompts still produced exact training continuations or identifiers.
Those facts can coexist. A rare, high-confidence disclosure barely moves an average over hundreds of ordinary documents. For the affected person, though, the average is not the harm.
The practical point is modest but important: a privacy audit should inspect the tail and report document-level extraction, not stop at one aggregate classification score.
Larger models leaked more in this model family
The reported identifier rate rose steadily across the Pythia series: 5.6% at 410 million parameters, 8.4% at one billion, 12.8% at 2.8 billion and 16.6% at 6.9 billion.
The distribution was uneven too. At 6.9 billion parameters, code, markup and configuration documents leaked at 35.8% in the 95-document subset. The rate in 405 prose documents was lower at 12.1%, but not zero.
A comparison between standard Pythia-2.8B and its deduplicated counterpart found 12.8% versus 14.0%. The difference was not statistically significant. That does not prove deduplication is useless: the test could miss a small benefit, and the two training runs differ in how often they pass over their data.
It does show that document-level near-deduplication, as implemented in this suite, should not be treated as a privacy guarantee.
The author has released the probe as an open-source command-line tool called leakit. Model Current has reviewed the paper and repository, but has not independently rerun the experiments.
What is confirmed, claimed and still open
Confirmed: the preprint, code and masked analysis records are public. The experiments cover Pythia models from 410 million to 6.9 billion parameters and OLMo-1B, using known training membership from public corpora.
The research finding: under the paper's partial-prefix setup, Pythia-6.9B reproduced an exact identifier for 83 of 500 targeted training documents after the context control. The rate rose with capacity in that model family and was higher in code-like material.
Still open: whether the result reproduces across seeds, newer open models, closed commercial systems, different corpora and less informed attackers. The paper is a single-author preprint and has not been peer-reviewed.
There is no basis here for saying that every model exposes 16.6% of its records. There is a basis for asking model developers to test for exact, per-document disclosure before release.
Averages are useful. They are just not where every privacy failure lives.
Sources
- Maricato - Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language ModelsPrimary CC BY 4.0 preprint submitted 31 July 2026. Source for the method, all reported extraction rates, controls, statistical intervals, limitations and ethics statement.
- Victor Maricato - leakit reproducibility repositoryPrimary code and analysis release linked by the paper. It publishes the sampling probe and masked records without raw identifiers or completions.
- Biderman et al. - Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingPrimary paper for the open model family used in the main capacity and deduplication comparisons.
- Duan et al. - Do Membership Inference Attacks Work on Large Language Models?Primary MIMIR benchmark paper used to contextualise the matched member and non-member evaluation split.
