Eight instructions left models with a 5.7% clean pass rate

One more instruction is not one more small risk

A model can follow each instruction reasonably often and still be poor at following all of them together. A new preprint puts a number on that gap.

The study tested 15 language models with combinations of rules about length, structure, wording, ordering and output format. At eight simultaneous constraints, the models passed an average individual rule 40.7% of the time. Only 5.7% of answers passed all eight.

That difference matters for real systems. A customer-support agent may need to answer the question, keep a tone, avoid private data, follow a policy, return valid JSON and stay within a length limit. None of those requirements is useful if one important rule quietly drops out.

The paper calls this constraint saturation. The loss on each rule was gradual. The chance of a completely clean answer fell much faster because the small failures accumulated.

The test counted rules, not impressions

Mariya I. Vasileva built the Constraint Saturation Evaluation around deterministic checks. A program, not another language model, decided whether an answer had the right word count, sentence pattern, required term, forbidden letter or data structure.

The benchmark covered 36 constraint types and 4,527 prompts, with between one and 12 compatible constraints placed into each prompt. It produced 369,753 checks across 15 models from eight families. The paper also included 444 deliberately impossible prompts to see which rule a model would sacrifice.

Across the satisfiable prompts, all 15 models showed the same broad shape. Complete success was 70.7% with one constraint, fell steeply through four to seven constraints and was close to zero from ten constraints onward in the aggregate.

The strongest model in the paper, GPT-5.5, stayed ahead by a wide margin. Even so, its all-pass rate fell below 50% when the prompt reached seven constraints. Twelve of the 15 models crossed that line at three constraints or fewer.

Some rules are easier to keep alive

The models did not lose every kind of instruction at the same speed. Structural rules degraded about twice as fast as lexical ones. Exact word counts, sentence counts and other requirements that need continuous tracking were particularly fragile.

A simple binary rule can survive because the model can satisfy it once. Include a required word, for example, and that job is done. Keeping every sentence inside a word range requires attention throughout the answer.

The researchers found that most failures were only weakly related to one another. There was no special combination of constraints that could be avoided to fix the problem. Each imperfect pass rate simply multiplied with the next.

That also explains an odd result in the impossible prompts. Models often preserved concrete, easy-to-see requirements and dropped abstract or structural ones. A response can look compliant on the surface while missing the rule that mattered more.

Prompt tricks bought little room

The paper tested three common fixes on three models. Asking for a plan before writing did not reliably move the threshold. A second self-correction pass pushed it by roughly one constraint in two cases and left one case unchanged. Taking the best of five attempts bought one or two additional constraints.

Those are useful gains, but not a cure. Retrying helps only when at least one answer happens to satisfy everything. It also adds latency and cost. A planning step can create another polished explanation without making the final output more exact.

For product teams, the practical response is less glamorous: count the obligations, test each one independently and use code for rules that code can verify. Long prompts should be treated as a bundle of failure probabilities, not a contract the model has fully understood.

Breaking a job into stages may help too, but this study did not test sequential chains. It measured rules that had to hold in one answer, all at once.

What is measured, inferred and still open

Confirmed: the single-author preprint was submitted on 12 August 2026. Its public manuscript describes 15 models, 36 constraint types, 4,527 prompts and deterministic scoring without an LLM judge.

The measured result: individual-rule performance declined gradually while all-rule success collapsed. At eight constraints, the aggregate rates were 40.7% for an individual rule and 5.7% for a completely compliant answer.

The author's interpretation: failures accumulate almost multiplicatively, and the main route to improvement is better reliability on each instruction rather than a clever ordering of instructions.

Important limits: the benchmark favours rules that software can check. It does not score factual accuracy, coherence, usefulness or tone consistency, and its 12-constraint prompts are deliberate stress tests. The intervention study covered only three models.

Still open: whether multi-step agent workflows fail in the same way, whether tool-based validators can hold the line in production and how newer training methods change the curve. For now, a long prompt is not one instruction. It is several chances to be wrong.