Skip to content
redactsure
Book a demo

Explore.

Data Report · By Chris Sowa · Published
Last updated

What are you doing to secure AI agents in production and how do you stop them from doing dumb stuff?

Production safety rests on unglamorous controls: least privilege identity, deterministic limits on tools, isolated and logged runs, and a named person approving anything irreversible. Without them, a steered, mistaken or overconfident model can finish an irreversible act on its own, and every ordinary error becomes an incident instead of a recoverable event.

A browser task passes permission and logging controls before a final action barrier.

Production safety rests on unglamorous controls: least privilege identity, deterministic limits on tools, isolated and logged runs, and a named person approving anything irreversible. Without them, a steered, mistaken or overconfident model can finish an irreversible act on its own, and every ordinary error becomes an incident instead of a recoverable event. Securing an AI agent in production therefore means assuming the model will sometimes be steered, mistaken, or overconfident, and building the system so that none of those states can produce an irreversible result alone. In practice that is a short list: give the agent an identity and the least privilege the task needs, keep sensitive values out of what the model reads where the task allows it, put deterministic checks and limits around its tools, require a named person to approve consequential actions under Supervised Delegation, isolate and log every run, and make stopping the agent easy. The second half of the question, stopping an agent from doing dumb stuff, is mostly not a prompt problem. Agents do avoidable damage when they have broad standing permissions, no volume limits, no confirmation step, and no one watching the record. Remove those conditions and most errors become recoverable events rather than incidents. The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company.

Key findings

  • OWASP lists excessive agency as LLM06 in the 2025 Top 10 for LLM Applications: an agent granted more functionality, permissions, or autonomy than necessary, with least privilege and human approval for sensitive actions as the core mitigations (OWASP, 2025).
  • Prompt injection, LLM01, and excessive agency combine in production. Injection supplies the bad instruction, excessive agency supplies the ability to carry it out (OWASP, 2025).
  • The NIST AI Risk Management Framework organizes this work as Govern, Map, Measure, and Manage, and treats it as a continuing life cycle activity rather than a launch checklist (NIST, 2023).
  • Joint guidance from CISA and partner agencies on agentic AI recommends layered controls with no single point of failure, strong identity management, human oversight mechanisms, fail safe defaults, adversarial testing, and documentation of agent actions (CISA et al., 2026).
  • Anthropic's published computer use guidance emphasizes keeping a human in the loop, isolating the environment, and limiting access, because a model operating a computer can be influenced by content on the screen (Anthropic, 2024).
  • OWASP lists sensitive information disclosure as LLM02, second only to prompt injection, and secondary copies such as screenshots, transcripts, traces, and logs are where an agent's handled values accumulate (OWASP, 2025).

What should be true before an agent reaches production?

Start with an inventory and an owner. Every production agent should have a named person accountable for it, a written purpose, a list of the systems and data it can reach, the tools it may call, and the actions it may never take. Agents without an owner keep running after a project ends, with permissions nobody reviews. The Map function of the NIST AI Risk Management Framework is the reference for this step: establish context, intended use, and risks before measuring anything (NIST, 2023).

Next, a threat model specific to the workflow. Which content does the agent read that a stranger can influence? Which values would cause harm if disclosed, changed, or sent to the wrong destination? Which action would be hardest to undo? Those answers decide where the controls below are strict and where they can be light.

Then test before exposure. Adversarial testing, evaluation across varied conditions and at different autonomy levels, and fail safe defaults are recommended in the joint agentic guidance before deployment (CISA et al., 2026). Testing should include injected content, malformed tool results, ambiguous instructions, and tasks the agent should refuse or escalate. Finally, decide the autonomy level explicitly and record it. Reading and drafting is a different risk from transacting, and many production problems come from an agent approved as an assistant that gradually acquired the ability to send, pay, and delete without a fresh decision.

How do you limit what an agent can do?

Least privilege is the highest value control, and it must be enforced outside the model. Give the agent its own identity, not a person's credentials and not a shared administrative account. Scope that identity to the specific applications, records, and operations the task requires, issue credentials for a short period, and let them expire when the run ends. Review access on a schedule and when the workflow changes.

Limit the tool set as strictly as the data set. An agent that can read a queue and draft a response does not need a delete function, a bulk export, or the ability to change permissions. Remove tools rather than instructing the model not to use them. Where a powerful tool is necessary, split it: reading, preparing, and committing should be separate steps, with the commit step gated.

Put hard limits in code around volume and value: maximum records per run, maximum amount per transaction and per day, rate limits, allowed destinations, and actions that are never permitted. OWASP guidance on unbounded consumption, LLM10, makes the related point for resource use: without limits, a loop or a steered agent can generate cost and volume indefinitely (OWASP, 2025). Isolate the run in a sandboxed environment with controlled network access, and keep secrets in a vault, injected at the point of use rather than placed in prompts or model context, where they can be repeated back or extracted.

How do you stop errors before they become actions?

Separate proposing from doing. The model may prepare an action: draft the message, assemble the payment, fill the form. Deterministic code then validates it against rules that do not depend on the model's judgment: Does the destination match the task? Is the amount inside the limit? Is the record the one the user named? Are required fields present and consistent with source data? Validation catches a large class of ordinary errors, not only attacks, because it checks the action rather than the reasoning.

Require approval for consequential actions, and make it usable. A named person should see what will happen, to what, for how much, and on whose instruction, in a form they can judge, and should be able to reject or edit. Approval requested so often that it becomes a reflex fails in practice. Group low risk steps, reserve approval for the irreversible ones, and never offer an always allow that silently converts a supervised agent into an unsupervised one.

Add confirmation for ambiguity as well as for risk. When an instruction matches two records, when source data conflicts, or when a task would exceed its budget, the agent should stop and ask rather than pick one. Many errors described as the agent doing something dumb are cases where a human would have noticed the ambiguity and the system gave the agent no way to surface it. Build in reversibility where the process allows it: drafts instead of sends, holds instead of payments, soft deletes with a recovery window. Where an action is truly irreversible, that alone argues for approval and a second check.

How do you keep sensitive data out of trouble?

Decide what the model needs to read, value by value, rather than giving it a whole screen or record. A model scheduling a payment needs an amount and a due date. It usually does not need the full account number, the customer tax identifier, or the unrelated history on the same screen. Minimizing what enters the context reduces both disclosure risk and the damage from a successful injection, even when the injection is never detected. Classification and loss prevention still have a role at outputs and destinations: check what an agent is about to send or write into a shared system, and block or route for review when it contains values that should not travel there. That is a second line, because by then the value was already in context.

Mind the secondary copies. Screenshots, transcripts, traces, and logs accumulate the values an agent handled, often with weaker access control than the source system. Redact or tokenize before recording where possible, set retention deliberately, and treat logs as sensitive data. OWASP lists sensitive information disclosure as LLM02, second only to prompt injection (OWASP, 2025). Control memory too: an agent that retains information between users or tasks can carry a value, or a planted instruction, into a context where it does not belong, so scope memory to the task and expire it.

How do you watch an agent and stop it when it misbehaves?

Log the run in a form an investigator can use: the instruction received, the sources read, the tool calls with their arguments, the validations applied, the approvals given and by whom, and the final state. A log recording only the final answer cannot support an incident review or a customer dispute. The joint agentic guidance recommends comprehensive artifacts documenting agent actions to support accountability (CISA et al., 2026).

Monitor for behavior, not only for errors. Signals worth alerting on include actions outside the usual set for that workflow, unusual destinations, volume or value spikes, repeated retries, and any attempt to use a tool the task does not require. Alert on attempted prohibited actions even when they were blocked, because a blocked attempt is evidence about the content the agent is reading.

Make stopping cheap and rehearsed. A kill switch that revokes the agent identity and credentials, pauses its queue, and preserves its logs should be a single, tested action owned by someone reachable. Test the stop path in a drill, and test rollback: how records changed by the agent are identified and corrected. Close the loop after every incident or near miss with a concrete change, such as a narrower permission, a new validation, or a regression test case, and track indicators such as blocked actions and escalations over time, as the Measure and Manage functions of the NIST framework suggest (NIST, 2023).

What makes agents do dumb stuff, and which habit prevents each cause?

Cause Typical production result Preventive habit
Broad standing permissions An error or injection reaches data and actions far beyond the task Per task least privilege, short lived credentials
No limits in code Loops, bulk changes, overspending, mass messages Caps on volume, value, rate, and destinations
Ambiguous instruction, no escalation path Agent guesses a record, a recipient, or an amount Stop and ask when a match is not unique
Untrusted content treated as instruction Planted text redirects the task Treat retrieved content as data, mark provenance, test with injections
Approval fatigue Reviewers approve without reading Reserve approval for consequential steps, present a clear summary
Secrets and sensitive values in context Values repeated, logged, or sent onward Minimize and tokenize before the model reads, redact logs
No record of the run Errors cannot be explained or undone Full action and approval record, exportable to the SIEM

None of these causes is a model quality problem. Each is a property of the surrounding system, which means it can be fixed without waiting for a better model.

Where does RedactSure sit?

RedactSure embodies that control set for browser based agent work. Supervised AI co-workers operate inside a governed virtual machine across an organization's existing applications, with no per-application integration and no change to user permissions. Policy selected values are replaced by consistent tokens through render-layer tokenization before the model reads any screen, so the model receives the task's fields with identifiers as ACCT_001 or SSN_001, and real values resolve only at approved destinations at the moment of an approved action. The Planner sets which values are tokenized and limits the permitted actions per screen, and a named person confirms that policy before the run.

There is no always allow, so the worst permitted action still pauses for the named approver, who approves every payment, submission and record change. The AI Control Record preserves the screens as tokens, the actions, and the approvals by name needed to reconstruct a bad run afterwards, and exports to the customer's own SIEM. Keys stay with the customer, and the model is swappable, so the controls do not move when the model does. Deployments are in pilot. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

This page rests on public guidance and standards, read as of October 13, 2026: the OWASP Top 10 for LLM Applications 2025, in particular the LLM01, LLM02, LLM06 and LLM10 entries (standard); the NIST AI Risk Management Framework 1.0 and the NIST adversarial machine learning taxonomy, AI 100-2e2023 (official guidance); the joint guidance on careful adoption of agentic AI services from CISA and partner agencies (official guidance); and Anthropic's computer use documentation and safety guidance (vendor documentation). The control set is drawn from what those documents recommend, and each control is paired with the failure it is meant to prevent.

The page describes a control set, not the results of an audit. No production system was inspected, no deployment was tested, and no hands-on testing of any named product was performed. No claim is made about how widely any control is implemented in practice or about the effectiveness of a specific product. Control effectiveness depends on implementation detail, and published guidance in this area is recent and still developing. Vendor features and agency guidance change, and the page carries its date and is revised when the sources change. The payment, queue and record examples describe task types, not any organization; no customer or prospect is described.

Specific recommendations should be checked against the workflow, data, and regulatory obligations of the organization applying them. Whether a control set meets those obligations is a determination for the organization's counsel, assessor, and its security and compliance officers, not for a research page.

What the record shows

A production agent will eventually be steered, mistaken, or overconfident, and the security question is whether any of those states can finish an irreversible act alone. The controls above share one design aim: they make errors cheap. Least privilege narrows what a bad run can reach, deterministic checks stop out of bounds actions, isolation and logging make each run inspectable, and named approval puts a person at the moment of commitment. The test for any production agent is one question: name the most damaging action it can complete without a person, and show the record that would explain it afterwards. If the answer is an irreversible act with no record, the agent is not secured, whatever its prompt says.

Frequently asked questions

Is better prompting the main fix for agent mistakes?

No. Stopping an agent from doing dumb stuff is mostly not a prompt problem. Prompts can be overridden by injected content, misunderstood under ambiguity, or followed too literally, and none of that is fixed by writing sterner instructions. The fixes that hold are structural: scoped permissions, limits on tools and volumes, deterministic checks before actions execute, and approval for consequential steps. Prompt quality affects how often the agent errs; the surrounding controls decide what an error costs.

Which single control prevents the most production damage?

Excessive agency is the risk to remove first, because it multiplies every other failure. OWASP lists it as LLM06 and names least privilege and human approval for sensitive actions as the core mitigations. Injection, misunderstanding, and plain model error all become more damaging as the agent's tools, permissions, and autonomy grow. Scoping the agent to the minimum its task needs, then gating what remains consequential, converts most incidents from losses into logged, recoverable events a team can learn from.

Why isolate and log every production run?

Isolation keeps one run's compromise or confusion from becoming the whole estate's problem, and published computer use guidance emphasises keeping the environment separate and access limited. Logging gives the isolation a memory: a complete record of what the agent saw, decided, and did is what lets a team stop a pattern, answer an auditor, or reconstruct an incident. Without the record, oversight is a claim. With it, oversight can be checked after the fact, run by run.

Should an agent have its own identity, or use the employee's credentials?

Its own identity. An agent running on a person's credentials inherits everything that person can reach, and a shared administrative account is worse. A dedicated identity can be scoped to the applications, records and operations the task requires, issued for a short period, expired when the run ends, and revoked in one step when something goes wrong. It also makes the log unambiguous about which actions were the agent's and which were the person's.

How do you avoid approval fatigue without removing the human?

Reserve approval for the consequential and irreversible steps, and group the low risk ones. Present each approval as a short, judgeable summary: what will happen, to what, for how much, on whose instruction, with the option to edit or reject. What not to do is add an always allow setting; that converts a supervised agent into an unsupervised one without a fresh decision, which is how assistants quietly acquire the ability to send, pay and delete.

What should an agent's kill switch actually do?

Revoke the agent's identity and credentials, pause its queue, and preserve its logs, in a single tested action owned by someone reachable. The stop path should be rehearsed in a drill, alongside rollback: how records the agent changed are identified and corrected. A kill switch that nobody has tried, or that depends on a person who is unavailable, is a plan rather than a control.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.