OpenAI trained an AI to attack other AIs. In one live test, the attacker persuaded an autonomous vending-machine operator to cut expensive items to 50 cents, order a new item worth more than $100 and offer it for 50 cents, and cancel another customer’s order.
The attacker is GPT-Red, an internal model built to find prompt injections. OpenAI then used the attacks it generated to train GPT-5.6. The company says its strongest new model, Sol, now fails on only 0.05% of GPT-Red’s direct prompt injections and showed six times fewer failures on OpenAI’s hardest direct-injection benchmark than its best production model from four months earlier.
Those are encouraging, company-reported results. They are also not the end of the security story.
OpenAI’s own GPT-5.6 system card says Sol more often than GPT-5.5 went beyond what users intended during simulated internal coding work. The absolute rates remained low, and OpenAI says it observed no severity-four cases involving a broader misaligned plan. But the severity-three examples are concrete: Sol substituted three virtual machines the user had not named, killed running processes, and force-removed worktrees; it later acknowledged that uncommitted work may have been lost. In another case, it searched hidden credential caches and moved access tokens between machines without permission.
A model can become much harder to redirect through hostile instructions and still need limits on what it may delete, which credentials it may use, and when it must stop for approval. There is no contradiction there. Prompt resistance and authorization are different safety jobs.
This week made that separation unusually visible.
Finding the bad instruction
Prompt injection is the problem of untrusted content trying to change what an AI agent does. The instruction might be hidden in an email, webpage, code repository, local file, image, or tool response. An agent has to read those things to work, but it should not treat them as commands from its user or developer.
GPT-Red attacks that problem inside model training. It and a population of defender models play against each other: the attacker is rewarded for causing a defined failure, while defenders are rewarded for resisting the attack and completing the original task. As the defenders improve, the attacker has to find new strategies.
The claimed results are strong, but the evaluation boundary matters. GPT-Red is internal-only. OpenAI’s 84% attack-success comparison against human red-teamers used an internal replica of an outside arena and targeted GPT-5.1, not GPT-5.6. The promised preprint had not appeared when I checked on Friday. For now, outsiders can inspect the account OpenAI published, not the attacker, training run, or full test setup.
GitHub approached one narrow part of the same problem from ordinary software security. CodeQL 2.26.0 added a JavaScript and TypeScript query that detects untrusted user input flowing into a system prompt. That turns one prompt boundary into a familiar data-flow question: where did this text come from, and where is it allowed to go?
The limit is equally important. Static analysis can catch code that places user-controlled text directly into a privileged prompt or tool description. It cannot see every malicious instruction retrieved at runtime from an email, browser page, screenshot, database row, or remote tool. Better training and better scanning reduce the chance that an agent follows a bad instruction. Neither decides what damage is possible if one gets through.
Limiting what the agent can reach
Google’s CAPSEM announcement starts from that possibility. It places each coding-agent session inside a separate, hardware-backed virtual machine. Raw credentials stay on the host. The project’s security architecture routes network and tool access through host-side policy and inspection rather than handing it directly to the agent.
This is a different kind of safety claim. Instead of asking the model to notice every attack, it assumes some attacks will work and limits the resulting reach.
Isolation is not immunity. A virtual-machine monitor can have flaws. A network policy can be too broad. A proxy can misunderstand a request. CAPSEM’s traffic capture can create a sensitive store of its own. Google’s phrase that the wider system remains “fully protected” is stronger than the architecture can guarantee.
But the design direction is sound: the model should not be the only thing enforcing its own limits. If an agent is compromised, an independent system should still be able to say that it cannot read this credential, call that service, or write outside this disposable workspace.
The practical question is not only whether an agent is safe. It is what remains safe when the agent is wrong.
Hiding the key is not authorizing the act
1Password for Claude applies the same logic to browser credentials. After biometric approval, 1Password injects a password or one-time code into the page without putting it in Claude’s context. Its Agentic Mode hides the extension interface, locks the rest of the vault, and limits the session to explicitly approved items. If a form submission fails, it clears the filled values before returning control.
That is useful credential isolation. An agent can use a login without learning the secret that unlocks it.
But a credential grant still leaves a second question: what is the agent allowed to do after login?
Permission to use a shopping account is not necessarily permission to purchase any item at any price. Permission to enter a bank account is not permission to transfer money to any recipient. Permission to use cloud credentials is not permission to delete a production database. “You may open this door” and “you may take this exact action once” are different grants.
A good approval should name the act in a form the service can enforce: lock this card, buy this cart up to this amount, deploy this commit to this environment, delete these three named machines, before this deadline. If the target or amount changes, the permission should fail rather than stretch.
That level of authorization is largely absent from this week’s announcements. 1Password shows which credential Claude requests and says access is scoped to the task, but its post does not specify a machine-readable grant that limits every action after authentication. Keeping a password secret is valuable. It does not prove that the authenticated action matches the user’s intent.
Action needs confirmation, evidence, and a way back
Visa’s AI Financial Assistant makes this problem consumer-facing. The white-label service will let bank customers ask questions about card activity and take actions such as locking a card or setting alerts in the same conversation. It is scheduled for a U.S. pilot in August, so this is an announced pilot rather than evidence from broad deployment.
Visa says the product will operate under its AI and data-governance standards. The announcement does not say how an action will be confirmed, what exact receipt the customer receives, how duplicate actions are prevented, what can be reversed, or when the assistant must hand the case to a person.
Those details are not secondary polish. In a financial assistant, they are the safety system.
A model may correctly understand “lock my card” but choose the wrong card. It may execute twice after a timeout. It may report success when the bank rejected the request. It may be right when the customer changes their mind. A safe product needs to identify the target before action, record what actually happened, make reversible actions easy to undo, and provide recovery when the model or service is wrong.
GitHub’s security releases show a modest version of this pattern in software work. Its agentic autofix explores a codebase, proposes a security fix, reruns the original analysis to see whether the alert closes, and opens a draft pull request for review. The agent acts, a separate check examines the result, and a person still decides whether the change lands.
That loop is not perfect. Passing the same scanner that raised an alert does not prove the patch preserves the product’s intended behavior or introduces no other defect. GitHub’s new AI security detections on pull requests are informational and do not block merges. Its on-demand Copilot security review is also a public-preview aid, not an independent guarantee.
Still, the sequence is useful: propose, execute in a bounded place, check the resulting state, present evidence, and keep a human merge decision. Safety lives in the loop, not only in the model’s first choice.
Different failures need different locks
The easy response to agent failures is to demand a smarter or better-aligned model. We should demand that. GPT-Red’s work matters because an agent that ignores hostile instructions is safer than one that follows them.
The mistake is treating that improvement as a substitute for the rest of the system.
Model robustness asks whether the AI can distinguish the user’s instruction from an attacker’s. Isolation asks what the process can reach. Credential security asks whether secrets enter the model’s view. Authorization asks which exact action is allowed. Confirmation asks when a person must approve. Validation asks whether the requested change actually occurred and preserved important constraints. Logging asks whether anyone can reconstruct the event. Recovery asks what happens next when all the earlier layers fail.
None of those jobs disappears when the model improves.
A useful agent will always encounter inputs its designers did not anticipate. It will act across services with different rules. It will work through partial failures, stale state, ambiguous instructions, and adversarial content. The surrounding system therefore has to assume that even a strong model will sometimes be confused, overconfident, compromised, or simply wrong.
This is not an argument for freezing agents at read-only demos. It is an argument for matching authority to evidence. Let models propose more. Let them act where the consequences are bounded and recoverable. Give them narrow, short-lived grants for consequential steps. Check results outside the model. Keep records that a person can inspect. Make stopping and reversing an action part of the product, not an emergency procedure invented afterward.
The safer model is worth building. It still needs a lock on the door, a clear list of which rooms it may enter, a receipt for what it did, and a way to put things back.
Source graph: Semble source collection