OpenAI built an AI attacker to make its agents harder to trick

An attacker built to teach the defender

OpenAI has trained an AI model whose job is to trick other AI systems.

It is called GPT-Red. The model looks for prompt injections: hidden instructions placed in an email, webpage, file or tool response that try to pull an agent away from the job its user gave it.

That risk grows when an AI can do more than answer a question. An agent may read private files, browse the web, write code or call business tools. If it treats untrusted content as an instruction, a short line buried in a document can become a route to a much larger action.

GPT-Red automates the work of finding those routes. It sends an attack, watches the target model respond and tries again. During training, the attacker and a collection of defender models improve together. GPT-Red receives a reward when it causes a defined failure. The defenders receive one when they resist the attack and still finish the original task.

As the defenders close an opening, the attacker has to find another one. This is the self-play loop behind OpenAI's announcement.

The idea is not to deploy GPT-Red as a product. OpenAI says it keeps the model internal and separate from its public systems because it was deliberately trained to produce effective attacks. The attacks it finds are instead fed into the training of later production models.

In plain terms, one model breaks the lock so another can learn how to hold it shut.

The test moved beyond a chat box

OpenAI did not limit GPT-Red to synthetic conversations.

In one case, the company pointed it at Vendy, an AI agent made by Andon Labs to manage a vending machine in an OpenAI office. GPT-Red first worked against a simulated version, then transferred its attacks to the live system.

OpenAI says it achieved three goals: changing the price of an expensive item to 50 cents, ordering a new item worth more than $100 and listing it for the same low price, and cancelling another customer's order. The company says the flaws were disclosed and new safeguards are being tested.

The vending machine is a modest system. That is partly why the example is useful. It turns prompt injection from an abstract chat problem into something with inventory, prices and actions that affect another person.

OpenAI also tested GPT-Red against a Codex command-line agent on ten data-exfiltration tasks. It says the specialised attacker was more successful and used fewer tokens than a prompted GPT-5.5 baseline. No full result table for that small test is public in the announcement.

These cases show what the model is intended to do. They do not show that every agent, or every deployment, is equally exposed.

The headline numbers need some care

OpenAI reports that GPT-Red found successful attacks in 84% of a set of novel scenarios against GPT-5.1. Human red-teamers succeeded in 13%.

The scenarios were based on a public prompt-injection competition described by Dziemian and 30 co-authors earlier this year. That competition drew 464 participants and 272,000 attack attempts across 13 frontier models. Every model was vulnerable, though the reported success rates varied widely.

For GPT-Red, OpenAI used an internal mirror of the competition and its own target model. That means the 84-to-13 comparison is a company result, not a fresh independent competition. The human and automated efforts may also differ in time, iteration budget and access to feedback. OpenAI has not provided enough detail to compare them cleanly.

The company reports a second set of gains on defence. It says GPT-5.6 Sol produced six times fewer failures on its hardest direct prompt-injection benchmark than its best production model four months earlier. Against GPT-Red's own direct attacks, the reported failure rate fell to 0.05%.

Those are striking figures. They are also narrow. A low rate on a held-out internal benchmark does not prove that a deployed agent is safe against a new attacker, a different tool setup or a long chain of untrusted content.

OpenAI said a technical preprint would follow later in the week. It was not available when this article was prepared, so the training recipe, evaluation budgets and full limitations cannot yet be checked.

A useful safety loop, with a sharp edge

Automated red teaming solves a real scaling problem. People are good at imagining abuse and noticing odd behaviour, but they cannot run millions of slightly different attacks during every model-training cycle.

A model can. It can also keep adapting when familiar benchmarks stop separating stronger systems from weaker ones.

There is a trade-off. Training a better attacker creates an asset that could be harmful if its methods or weights escaped. Keeping GPT-Red closed may reduce that risk, but it also makes independent examination harder. Outside researchers can test the released product models. They cannot inspect the attacker that shaped them.

The loop can also become too self-referential. A defender trained against one powerful internal opponent may look excellent against that opponent while remaining weak in a different environment. Human red teams, public competitions and third-party testing still matter because they bring different assumptions and incentives.

OpenAI says it will continue using those layers alongside GPT-Red. The public evidence is not yet detailed enough to judge how the layers fit together.

What this changes for people using agents

GPT-Red is not a reason to treat prompt injection as solved.

It is evidence that model makers are putting much more compute into the problem, and that attacks can now be generated at roughly the same kind of scale as training data. That should improve the base model. It does not remove the need for controls around it.

An agent should still get only the tools and data it needs. Sensitive actions should have clear approval points. Untrusted documents should not be allowed to silently redefine a task. Logs need to show what the agent read, what it decided and which tool it called.

These measures sound less impressive than self-improving red teams. They are what limit the damage when a model does follow the wrong instruction.

The most encouraging part of GPT-Red is the direction: safety testing is moving from a one-off exercise into a continuous contest. The unresolved part is equally important. Most of the evidence comes from the company that built both the attacker and the defender.

The promised paper should make that contest easier to understand. Independent tests will tell us whether the stronger door holds when someone else chooses where to push.