A better AI model can still lose answers the old one got right

The average went up. Some answers still went backward

A model update can improve every headline benchmark and still break the exact question a team cares about.

Researchers at the University of Ottawa and the Vector Institute studied this smaller kind of failure. They call it a negative flip: the old model answered correctly, then the new model answered the same item incorrectly.

Across six model update pairs and six benchmarks, the newer version always improved average accuracy. Negative flips still appeared in multiple-choice questions, maths problems and code generation.

The practical problem is not just finding that regression after release. It is spotting the risky request while both versions are available, before the new answer is trusted.

The test compared six real version changes

The study used open-weight instruction models from Qwen, Llama, Mistral and Gemma. Five updates stayed within one family. A larger Llama 2 to Llama 3 jump was treated as a boundary case.

The benchmarks covered multiple-choice knowledge with MMLU-Pro and GPQA, maths with GSM8K and MATH, and code generation with HumanEval and MBPP. Answers were generated once with deterministic decoding.

The researchers then tested signals available at inference time. Some came from one model, including confidence, logit margin and attention entropy. Others compared both versions through changes in output probabilities, token likelihoods or internal representations.

This was a white-box study. Several signals need logits, hidden states or attention weights that a closed model API may not expose.

Confidence was useful, just not everywhere

For multiple-choice questions, the new model's confidence usually carried most of the useful warning. Correctness there depends on one discrete choice, so uncertainty around that choice is a sensible signal.

Harder generative work behaved differently. On the full MATH set and the two coding benchmarks, changes in likelihood or token distributions between versions added useful information more often. A completion can sound confident and still fail a unit test.

Even that pattern was not universal. The strongest signal changed with the task and sometimes with the particular model pair. Combining several cross-version signals rarely improved on the best single one.

There is no clean dashboard light that turns red whenever a model update is about to lose an old capability. The warning system has to match the work being done.

Keeping the old model is not an automatic safety net

The paper tested a simple fallback idea. Rank requests by estimated regression risk, then send the riskiest 10% back to the previous model.

The better trigger changed by update. Confidence caught more regressions on one Qwen maths test. A cross-version signal caught more on Llama code and Gemma code.

Yet catching a regression is only half of the trade-off. Returning to the old model can also discard an answer the new model fixed. The fallback delivered a clear net accuracy gain in only one tested case, adding 1.8 percentage points on Gemma HumanEval.

That makes rollback more like a calibrated product decision than a universal guardrail. A team needs representative tasks, both versions and a definition of which mistakes matter most.

What is confirmed, found and still uncertain

Confirmed: the two-author preprint was submitted on 11 August 2026. It covers six model update pairs, six benchmarks and three task families. The authors released their analysis code.

Found in this setup: higher average accuracy did not prevent sample-level regressions. Confidence was strongest on multiple-choice and simpler maths, while cross-version likelihood signals helped more often on harder maths and code. No signal dominated everywhere.

The researchers' interpretation: output structure helps explain the split. One-choice answers expose uncertainty differently from long reasoning or executable code. They present that explanation as a hypothesis, not a proved mechanism.

Important limits: the models had roughly 7 to 9 billion parameters and were run without an explicit reasoning phase. Code benchmarks were small, one response was sampled per item and the strongest signals require internal model access.

Still open: how the method behaves with closed frontier APIs, larger code suites, agent scaffolds or system-prompt changes. A model upgrade is not one number. It is a new pattern of wins and losses.