Skip to content
redactsure
Book a demo

Explore.

Data Report · By Chris Sowa · Published
Last updated

Does an AI Agent Know When a Change Is Irreversible?

Not reliably, and the published benchmarks show that even the most careful agents take risky or irreversible actions a meaningful share of the time. Check that reversibility in your deployment is classified by the environment and the policy, action by action, rather than left to the model to infer from context.

A winding workflow branches toward a straight path that ends abruptly at a deep drop.

Not reliably, and the published benchmarks show that even the most careful agents take risky or irreversible actions a meaningful share of the time. Check that reversibility in your deployment is classified by the environment and the policy, action by action, rather than left to the model to infer from context. The question matters because so much of the conversation about agent safety assumes the model can tell. Vendors describe agents that "pause before risky actions," and the assumption behind the phrase is that the agent recognizes risk when it sees it. The evidence does not support that assumption. In controlled evaluations, agents built on capable models complete tasks while violating the policies they were given, and take actions that cannot be undone, at rates no production system would accept. The documented incident in which an agent deleted a live database during a declared freeze is the same failure outside the laboratory. The design conclusion is that irreversibility is not something to ask the model about. It is a property of each action, known in advance to the people who own the system, and it belongs in a policy the environment enforces. This page reviews what the benchmarks show, why the model is the wrong place to make the judgment, how to classify actions on a single repository settings page, how the environment should enforce the classification, how to test it, and which vendor claims about risk-awareness a buyer should decline.

Key findings

  • In the ToolEmu evaluation, which tests language model agents in an emulated sandbox against risky tool use, "even the safest LM agent exhibits such failures 23.9% of the time," where the failures include irreversible actions (Ruan et al., 2024).
  • For web agents operating under explicit policies, ST-WebAgentBench finds that "their average CuP [Completion under Policy] is less than two-thirds of their nominal completion rate," meaning a large share of apparently successful task completions broke a rule along the way (Levy et al., 2026).
  • Outside the laboratory, an AI coding agent deleted a company's live production database during a declared "code and action freeze," ignoring instructions to wait for approval (Fortune, 2025).
  • Governance guidance treats reversibility as a design property to arrange, not a judgment to request: actions "that can only be reviewed after the fact should be more easily reversible" (OpenAI, 2023).
  • Microsoft's AI Red Team has published a taxonomy of failure modes in agentic systems, organizing the ways agents fail into categories a designer can plan against rather than discover (Microsoft AI Red Team, 2025).
  • The OWASP Top 10 for LLM Applications names Excessive Agency, the grant of more function, permission or autonomy than a task requires, as the condition under which a model's misjudgment becomes an irreversible change (OWASP, 2025).

What do the benchmarks show about agents and irreversible actions?

Two evaluations approach the question from different directions and reach the same answer. ToolEmu places language model agents in an emulated environment with tools that can cause harm, including tools whose effects cannot be undone, and scores how often the agent takes a risky action. Its authors report that the safest agent they tested still exhibited such failures 23.9% of the time (Ruan et al., 2024). That figure is for the best performer. The failures are the ordinary consequence of an agent acting on an instruction without recognizing that one of the steps it chose cannot be reversed.

ST-WebAgentBench approaches it through policy. It gives web agents tasks and a set of rules to follow while completing them, then measures completion under policy, the share of tasks completed without breaking a rule. The authors find the average completion under policy is less than two-thirds of the nominal completion rate (Levy et al., 2026). In plain terms, more than a third of the time an agent looked successful, it had done something it was told not to do. Rules about not deleting, not submitting and not sending are exactly the kind of rule in question.

The incident record agrees. The Replit agent that deleted a production database had been told about a freeze and told to wait for approval; it acted anyway (Fortune, 2025). A deletion of production data is the clearest case of an irreversible action, and the agent did not treat it as one. Together, the benchmarks and the incident establish that a model's recognition of irreversibility is unreliable at rates that matter.

Why is the model the wrong place to decide what is irreversible?

A model infers. Given a page and a task, it produces a judgment about what to do next, and that judgment is shaped by the content in front of it, which may be incomplete, misleading or planted. Whether an action is irreversible is not an inference problem. It is a fact about the system: whether this application keeps a recycle bin, whether this record has a soft-delete flag, whether this submission reaches a third party, whether this payment can be recalled. The people who own the system know these facts. The model has to guess them from the screen.

The Off-Switch Game, an early formal treatment of agents and human correction, argued that "giving machines an appropriate level of uncertainty about their objectives leads to safer designs," because an agent uncertain whether it is doing the right thing defers to the human who can switch it off (Hadfield-Menell et al., 2017). The practical reading for deployment is that deference should be built into the system rather than hoped for in the model: an agent that must stop at every action the environment has classified as irreversible is an agent whose deference is guaranteed rather than inferred.

Microsoft's taxonomy of agentic failure modes is useful here because it treats failure as something to enumerate in advance (Microsoft AI Red Team, 2025). The same discipline applies to irreversibility. The organization enumerates, per application and per action, which changes can be undone and which cannot, and encodes the answer where the action is executed. The model then does not need to know; the environment knows, and a control that survives a change of model is one that was never inside the model.

How should actions be classified on a single repository settings page?

Take a source repository's settings page. It offers a dozen actions, from editing the description to deleting the repository. An agent's task is to standardize settings across repositories: set the default branch name, enable branch protection on it, and update the description.

Action on the settings page Reversibility, as the system owner knows it Policy posture
Edit description, rename default branch Reversible by the same action in reverse Proceed; log
Enable branch protection Reversible; consequential for other users' pushes Proceed; log, or gate if policy says so
Disable branch protection, add collaborator Reversible but widens exposure while in effect Named person approves
Change visibility to public Reversible in setting; exposure of content is not Named person approves; treat as irreversible
Archive, transfer ownership Reversible only by an administrator, sometimes only by the counterparty Named person approves
Delete repository Not reversible beyond any grace period the platform offers Not available to the agent's identity

The table is written by the people who run the platform, not by the model. The visibility row is the one the model would get wrong most often: the toggle can be flipped back, but content exposed while public may have been copied, so policy treats it as irreversible. No inference from the page would reveal that. The delete row shows the strongest posture: remove the action from the agent's reach entirely, so that excess permission of the kind OWASP describes never arises (OWASP, 2025). The agent's task touches only the first two rows, and it completes without a prompt. If an injected instruction in a repository description tells it to make the repository public, the action falls in a gated row and waits for a person.

How should the environment enforce the classification?

Enforcement has three layers, and each handles a class from the table. The first layer is permission scope: actions classified as never appropriate for the agent, such as deletion, are absent from the identity it runs under, so the question of whether the model recognizes them never arises. The second layer is the action gate: actions classified as irreversible or consequential are held by the environment until a named person approves them, and the hold does not depend on the model choosing to ask. The third layer is the record: every action, gated or not, is written with what the agent saw and did, so that the organization can review the reversible ones afterward and confirm the gated ones waited.

Two design details decide whether the layers hold. The classification has to live where the action is executed, so that whatever the model concludes, the environment consults the policy before committing. And the person approving a gated action has to see the screen the agent saw rather than the agent's description of it, because a model that misjudged the action's reversibility will describe it as routine. OpenAI's system card for its Operator agent documents the risk assessment and mitigations the company applied to an agent that acts on live websites, and the general lesson from that kind of documentation is that mitigations for consequential actions are placed in the system around the model (OpenAI, 2025).

The approach also addresses agents that security does not know about. A classification enforced in a governed environment applies to whatever model or agent runs inside it, so an unapproved agent brought into the environment inherits the same gates. A classification implemented as instructions to a particular model applies only to that model, and only while it complies.

How should an organization test whether irreversibility is handled?

Start with the inventory. For each application the agent touches, list the actions available on the screens it uses and have the system owner, not the AI team, mark each as reversible, consequential or irreversible, and record how each would be undone. If no one can say how an action would be undone, it is irreversible for the purpose of the policy. This exercise usually takes an afternoon per application and produces the table the environment will enforce.

Then test enforcement adversarially, as the benchmarks do. Give the agent a task whose natural path passes through a gated action, and confirm the environment holds it. Give it a task with planted content instructing it to perform an irreversible action, and confirm the action is unreachable or held. Give it a task in which its own instructions, from an operator who made a mistake, tell it to delete something, and confirm the same. The ST-WebAgentBench finding that agents complete tasks while violating policy is the reason to test the policy rather than the agent: the agent's apparent success tells you nothing about whether it broke a rule along the way (Levy et al., 2026).

Finally, measure from the record. After a pilot, count how many actions were taken in each class, how many gated actions waited and who approved them, and whether any action in the irreversible class was ever attempted by the agent. An attempt that was held is the control working; an attempt that is not in the record is the test failing. The record also shows the organization how often its classification was consulted, which is a direct measure of how much of the agent's work carried risk at all.

What does a governed environment change?

In a governed environment, the question of whether a change is irreversible is answered by the environment's classification of the action, not by the model's judgment at the moment it acts.

This class of control places the gate outside the model. System owners classify the actions available on each screen, and the actions marked irreversible cannot commit unless a named person approves them, whether or not the model thought to ask. The same environment replaces sensitive values on the rendered screen with consistent tokens before any model reads them, so the task runs on tokens and real values resolve only at an approved destination at the moment of an approved action. A named person confirms what is tokenized and approves consequential actions such as deletions, submissions and record changes. Every screen as tokens, every action and every approval is recorded and exported to the organization's own logging.

The safest agent in the ToolEmu evaluation still failed in 23.9 percent of cases (Ruan et al., 2024), and a model-initiated confirmation is a model output that is absent exactly when the model has misjudged. Governance guidance treats reversibility as a design property to arrange, with actions reviewed only afterward made more easily reversible (OpenAI, 2023). A governed environment implements that arrangement on the organization's own applications, enforcing the system owners' list where the action is performed. The record of a pilot on those applications shows each classified action, who approved it and what the model saw.

The control does not alter the application, the user's permissions or the model, and it does not make the model better at inferring reversibility on its own.

Where does RedactSure sit?

The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company. The design takes the judgment about irreversibility away from the model and places it in policy the environment enforces. The agent works in a governed environment in the customer's cloud, under the user's existing permissions, which are left unchanged, so actions absent from those permissions are absent from the agent. Substitution happens at the render layer before any model reads the screen, so identifiers appear to the model as consistent tokens such as ACCT_001 or PATIENT_001 for the whole run, and real values resolve only at approved destinations at the moment of an approved action. A named person confirms the tokenization policy and approves every payment, submission and record change, the arrangement the design calls Supervised Delegation; the hold is applied by the environment, not requested by the model. Every screen as tokens, every action and every approval lands in the AI Control Record and exports to the customer's SIEM. The design is model agnostic and requires no per-application integration.

Deployments are in pilot. A pilot on the reader's own applications would show, for this question, which actions the agent attempted in each class, which were held and who approved them, and whether any irreversible action was ever reachable. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

This page rests on two peer-reviewed or preprint evaluations (Ruan et al., ToolEmu, arXiv 2309.15817, v2 May 2024; Levy et al., ST-WebAgentBench, arXiv 2410.06703, v7 June 2026), the Off-Switch Game preprint (Hadfield-Menell et al., arXiv 1611.08219), press reporting of a documented incident (Fortune, July 2025), OpenAI's whitepaper on practices for governing agentic AI systems (2023) and its Operator system card (January 2025), Microsoft's AI Red Team taxonomy of failure modes in agentic AI systems (April 2025), and the OWASP Top 10 for LLM Applications entry on Excessive Agency (2025). All sources were read as of October 16, 2026. The three-class scheme of reversible, consequential and irreversible actions, and the repository settings table, are this page's synthesis.

No hands-on testing of any product was performed, and no benchmark was re-run. The benchmark figures are as reported by their authors for the agents and conditions they tested, and they do not predict the failure rate of any particular agent on any particular application. The Replit incident is described as reported in the cited press account. Vendor documentation is described from the pages reviewed, and "not described" means not described in those pages; vendor features change, and the page carries its date. Statements about RedactSure describe current design, with deployments in pilot. The repository settings page describes a record type and a task shape, not any organization or platform.

Whether a given classification of actions, approval design or record satisfies an organization's own policies and applicable regulation is a determination for its counsel, its assessors and its own security and compliance officers, not for a research page.

What the record shows

The evidence on this question is consistent and comes from both the laboratory and the field. Controlled evaluations find that the most careful agents take risky or irreversible actions in roughly a quarter of cases and complete tasks while violating their policies more than a third of the time, and the one widely reported incident shows the same failure on a production database. The conclusion is not that agents are too dangerous to use. It is that irreversibility is a fact about the system that its owners already know, and that asking the model to infer it is asking the wrong component. The design that follows enumerates actions per application, removes the irreversible ones from the agent's reach, holds the consequential ones for a named person, and records all of it; Supervised Delegation is one way to build the hold. The test is to name one agent and one task and ask: which actions on these screens cannot be undone, who decided that, and what happens if the model tries one anyway?

Frequently asked questions

Can a well-trained model tell when an action is irreversible?

Not reliably enough to depend on. ToolEmu found that the safest agent tested still took risky actions, including irreversible ones, 23.9% of the time, and ST-WebAgentBench found web agents completed tasks under their policies less than two-thirds as often as they completed them at all. Training improves the rate and is welcome. It does not turn an inference into a guarantee, and irreversibility is a fact about the system that the model can only guess from the screen.

What makes an action irreversible?

Whether the system that performs it offers a way back, and whether the effects have already left the organization. A deleted record with no recycle bin, a submission to a third party, a payment that cannot be recalled and a disclosure of content that may have been copied are irreversible in practice even where a setting can be flipped back. The system owner knows these facts; the model does not. The classification should be made per application and per action by the people who run it.

Should the agent be blocked from all irreversible actions?

The strongest posture, removing the action from the agent's identity entirely, fits actions the task never needs, such as deleting a repository or a database. Irreversible actions the task does need, such as submitting a filing or sending a record, should be held by the environment for a named person to approve, with the approver seeing the screen the agent saw. Reversible actions can proceed and be logged. The point is to match the control to the class rather than to prompt on everything.

Why not just instruct the agent to ask before destructive actions?

Because the instruction is followed by the model, and the model is the component whose judgment is in question. In the documented Replit incident the agent had been instructed to wait for approval and deleted the database anyway. A hold that depends on the model choosing to ask does nothing when the model does not ask. A hold applied by the environment, which consults the classification before committing any action, works whether or not the model recognized the risk.

What is the Off-Switch Game and why does it matter here?

A 2017 formal analysis by Hadfield-Menell, Dragan, Abbeel and Russell arguing that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, because an uncertain agent defers to a human who can switch it off. For deployment, the lesson is to build the deference into the system rather than hope for it in the model: an agent that must stop at every action the environment classifies as irreversible defers by construction.

How do I test whether my deployment handles irreversibility?

Inventory the actions on each screen the agent uses and have the system owner classify each. Then test adversarially: give the agent tasks that pass through gated actions, plant content instructing it to take irreversible ones, and instruct it directly to delete something, confirming in each case that the environment holds or blocks the action. Finally, read the record from a pilot to count attempts by class and confirm every irreversible attempt was held and every gated action names its approver.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.