Skip to content
redactsure
Book a demo

Explore.

Data Report · By Chris Sowa · Published
Last updated

How Likely Is an AI Agent to Leak Data?

Likely enough to plan for: peer-reviewed benchmarks put inadvertent leakage by language model agents between roughly a quarter and over half of tested cases, even with privacy instructions, and targeted attacks on web agents reach 70 percent. Treat leakage as a base rate to engineer around and measure your own agents against it.

A stream of customer records passes through layered walls while small fragments escape beyond the final boundary.

Likely enough to plan for: peer-reviewed benchmarks put inadvertent leakage by language model agents between roughly a quarter and over half of tested cases, even with privacy instructions, and targeted attacks on web agents reach 70 percent. Treat leakage as a base rate to engineer around and measure your own agents against it. The question is usually asked as if the answer were a matter of opinion, with vendors saying rarely and skeptics saying constantly. It is in fact a measured quantity. Since 2023, several research groups have built benchmarks that place a language model in a situation where it holds sensitive information and has a reason to share some of it, then count how often it shares what it should not. The numbers are consistent across groups, models and years, and they are high. They are also, importantly, numbers for the inadvertent case: the model with no attacker, trying to be helpful. The adversarial case, where someone has planted instructions in content the agent reads, is worse. This page sets out what the benchmarks measure and find, whether acting changes the rate, how much an attacker adds, why instructions do not bring the rate down enough, what a leak costs and how to set a rate for one's own agents, and which claims a buyer should refuse. It builds on the definition of AI agent data exposure and the broader frame of AI agent data security.

Key findings

  • In the PrivacyLens benchmark, agents built on GPT-4 and Llama-3-70B leaked sensitive information in 25.68 percent and 38.69 percent of cases respectively, even when given privacy-enhancing prompts (Shao et al., 2024).
  • Tested against contextual integrity norms, GPT-4 and ChatGPT revealed private information in contexts where humans would not in 39 percent and 57 percent of cases (Mireshghallah et al., 2024).
  • Web agents built on GPT-4, Llama-3 and Claude were found prone to inadvertent use of unnecessary sensitive information when completing tasks, which is leakage through action rather than through conversation (Zharmagambetov et al., 2025).
  • Under attack, the rate rises: the Environmental Injection Attack, which hides instructions in page elements, achieved up to 70 percent success in stealing specific personal information from generalist web agents (Liao et al., 2025).
  • Web-based indirect prompt injection against AI agents has been observed in the wild, not only in laboratories, which moves the adversarial rate from hypothetical to operational (Palo Alto Networks Unit 42, 2026).
  • Sensitive information disclosure is a named category in the OWASP Top 10 for LLM applications, which treats it as an expected failure mode to be designed against rather than an anomaly (OWASP, 2025).

What do the benchmarks measure, and what do they find?

Take a loan application record: applicant name, social security number, income, employer, credit score, the loan officer's notes, and a correspondence thread with the applicant. An agent assisting the loan officer is asked to draft a status update for a third party, such as a real estate agent waiting on the approval. The task requires saying whether the loan is on track. It does not require the credit score, the income or the officer's notes, all of which are in the agent's context.

The privacy benchmarks construct this situation and count what the model does. The contextual integrity work builds scenarios in which information is appropriate in one context and not another; GPT-4 and ChatGPT revealed private information in contexts where humans would not in 39 percent and 57 percent of cases (Mireshghallah et al., 2024). PrivacyLens extends the method to agents that act, seeding the context with details the task does not need; GPT-4 and Llama-3-70B agents leaked them in 25.68 percent and 38.69 percent of cases even with privacy-enhancing prompts (Shao et al., 2024).

Benchmark Setting Reported leakage
Contextual integrity tests Model shares information across contexts with different norms 39 percent (GPT-4) and 57 percent (ChatGPT)
PrivacyLens Agent completes tasks with unneeded sensitive details in context, privacy prompts applied 25.68 percent (GPT-4) and 38.69 percent (Llama-3-70B)
AgentDAM Web agents complete realistic tasks with unnecessary sensitive data available Agents on GPT-4, Llama-3 and Claude prone to inadvertent use of it
EIA Web agents under hidden-element injection attack Up to 70 percent success in stealing specific personal information

For the loan agent, the finding is that in roughly one draft in three to one in two, depending on the model, the status update to the real estate agent will contain something it should not.

Does the rate change when the agent acts rather than chats?

The earlier benchmarks tested conversation: a model asked a question, answering with too much. Agents raise a different concern, because an agent does not only answer, it fills forms, sends messages and updates records, and each of those is a channel through which information can leave. AgentDAM was built to measure this. It places web agents in realistic tasks where sensitive information is available but unnecessary, and finds that agents built on GPT-4, Llama-3 and Claude are prone to inadvertent use of unnecessary sensitive information in completing the task (Zharmagambetov et al., 2025).

The mechanism is not malice or confusion. It is helpfulness operating on a context that contains more than the task needs. The loan agent drafting the status update has the credit score in front of it, and including it makes the update more informative. Nothing in the model's objective distinguishes informative from appropriate. The privacy benchmarks are, in this sense, measuring how often a model's drive to be useful overrides a norm it was never given a reliable way to apply.

Acting also compounds the consequence. A chat response leaks to the person asking. An agent's form submission leaks to whatever system receives the form, and an agent's email leaks to whatever address it chose, which may be one it read off a page. The rate per decision may be similar between chat and action; the surface area per leak is not. An agent that can navigate screens and act on them has, by construction, more places for a leaked value to go than a model that can only reply.

How much does an attacker add to the base rate?

The figures above are for the inadvertent case, with nobody trying to cause a leak. When someone is, the picture changes. The Environmental Injection Attack hides instructions in elements of a web page that a person would not notice but an agent reads, and directs the agent to disclose specific personal information; the attack achieved up to 70 percent success in stealing specific personal information from generalist web agents (Liao et al., 2025). The agent in that study was not confused about its task. It was reading a page that told it to do something else, and it complied most of the time.

This is the indirect prompt injection pattern first demonstrated against deployed LLM-integrated applications in 2023, in which retrieved content redirects the model's behavior without the user's knowledge (Greshake et al., 2023). It has since left the laboratory. Palo Alto Networks' Unit 42 reports web-based indirect prompt injection against AI agents observed in the wild, which means the adversarial rate is now an operational concern for any agent that reads content it did not author (Palo Alto Networks Unit 42, 2026).

For the loan agent, the attacker's path is the correspondence thread. An applicant, or someone who has compromised an applicant's email, includes an instruction in a message the agent will read while drafting its update. The agent now has a 70 percent class of risk layered on a 30 percent class of risk, and the two are not alternatives. An organization planning for the inadvertent rate alone is planning for the quiet day.

Why do privacy instructions not bring the rate down enough?

The intuitive fix is to tell the model not to leak. The benchmarks tested that. PrivacyLens applied privacy-enhancing prompts and still measured 25.68 and 38.69 percent leakage (Shao et al., 2024). The instruction helps at the margin and leaves the rate at a level no one would accept from a human employee. The reason is structural: an instruction is one more piece of text in a context full of text, and the model weighs it against the task, the content it is reading and whatever else is present. Under injection, the attacker's instruction is in the same context competing for the same weight.

OWASP's Top 10 for LLM applications lists sensitive information disclosure as a standing category, with the implication that it is to be designed against at the system level rather than prompted away (OWASP, 2025). The International AI Safety Report's 2025 key update, surveying capabilities and their risk implications, places these failure modes in the same frame: properties of current systems to be managed, not defects awaiting a patch (International AI Safety Report, 2025).

What does bring the rate down is removing the information from the context. A model cannot leak a credit score it never read. If the loan record arrives at the model with the score, the income and the identifiers as tokens, the inadvertent rate for those fields is zero and the adversarial rate for those fields is zero, whatever the model does. Masking sensitive data before the model reads it converts a behavioral problem, which the benchmarks show is not solved, into an architectural one, which is.

What does a leak cost, and how should an organization set its own rate?

IBM's annual Cost of a Data Breach Report measures what organizations actually pay when data leaves, across industries and causes, and is the standard reference for the financial consequence of the event the benchmarks count (IBM, 2025). The relevant arithmetic for a planning exercise is simple: the number of agent decisions per day that touch sensitive data, multiplied by a leakage rate in the published range, is the expected number of leak events per day before any attacker is considered. For an agent handling a few hundred loan files a week, a rate of one in four is not a tail risk. It is a Tuesday.

The published rates are not the organization's rate, and the way to find that is to measure it. Take one agent and one task, with real record types and real screens. Seed the context the way the benchmarks do, with sensitive fields the task does not need. Run the task repeatedly and count the outputs that contain any of them. Then plant an instruction in content the agent reads and count again. The two counts are the organization's inadvertent and adversarial rates for that agent, and they are the numbers a risk committee can act on.

Then run the same measurement with the sensitive fields tokenized before the model reads them. The count for those fields should be zero in both conditions, and the difference between the two runs is the value of the control, stated in the organization's own units. Preventing leaks without blocking legitimate work is the constraint on that design: the task must still complete, which the second run also shows.

What does a governed environment change?

In a governed environment, the leakage rate stops being a property of the model's behavior and becomes a property of what the model was given, because a value the model never read in the clear cannot be leaked by it at any rate.

The mechanism is a class of control between the rendered screen and the model. Sensitive values on the page, including names, account numbers, incomes and the free-text notes a filter would miss, are replaced with consistent tokens before any model reads them, and the task runs on those tokens. Real values resolve only at an approved destination at the moment of an approved action. A named person confirms what is tokenized and approves consequential actions, including payments, submissions and record changes. Every screen as tokens, every action and every approval is recorded and exported to the organization's own logging.

This bears on the title question because the benchmarks measured the alternative. Agents given privacy-enhancing prompts still leaked in 25.68 and 38.69 percent of cases (Shao et al., 2024), and hidden page instructions extracted specific personal information from web agents with up to 70 percent success (Liao et al., 2025). Both figures describe a model that held the real values. The environment changes the quantity being measured: an attacker who succeeds against the model obtains tokens, and the organization's own rate, inadvertent and adversarial, can be read from the run record field by field rather than taken on assurance.

The control does not change the model or the application, and it does not lower the model's own disclosure tendency; it changes what is available to disclose.

Where does RedactSure sit?

The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company. The design treats the benchmark rates as given and removes what they act on. A governed environment in the customer's cloud substitutes sensitive values at the render layer before any model reads the screen, with consistent tokens such as ACCT_001 or PATIENT_001 for the whole run, so the loan record's identifiers, score and income arrive at the model as tokens and the inadvertent and adversarial leakage rates for those fields do not depend on the model's behavior. Real values resolve only at approved destinations at the moment of an approved action. A named person confirms the tokenization policy and approves every payment, submission and record change, which the company calls Supervised Delegation. Every screen as tokens, every action and every approval lands in the AI Control Record and exports to the customer's SIEM. The environment is model agnostic, requires no per-application integration and leaves user permissions unchanged.

Deployments are in pilot. On the reader's own agent, a pilot would produce the two counts this page describes, inadvertent and under injection, with and without tokenization, on real screens. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

This page rests on peer-reviewed and preprint benchmarks of privacy behavior in language models and agents: Shao et al. on PrivacyLens (NeurIPS 2024 Datasets and Benchmarks), Mireshghallah et al. on contextual integrity (2024), Zharmagambetov et al. on AgentDAM (2025), Liao et al. on the Environmental Injection Attack (2025) and Greshake et al. on indirect prompt injection (2023); on Palo Alto Networks Unit 42's report of injection observed in the wild (2026; industry analysis); on IBM's Cost of a Data Breach Report (2025; industry analysis); on the International AI Safety Report's first key update (2025); on OWASP's LLM02:2025 entry (standard); and on RedactSure research pages. All sources were read as of October 16, 2026.

No hands-on testing of any model or product was performed. The percentages quoted are the figures the cited papers report for the specific models, versions and task sets they tested, at the dates they tested them; newer model versions may score differently, and no figure on this page is a prediction for any organization's agents. The benchmarks measure leakage as each paper defines it, and definitions differ between them, so the figures are a range rather than a single rate. Vendor features change, and the page carries its date. Statements about RedactSure describe its current design, with deployments in pilot. The loan application record used as a worked example describes a record type, not any lender or applicant.

Whether a measured leakage rate, a tokenization policy or an approval design meets an organization's obligations is a determination for its counsel, its assessors and its own security and compliance officers. This page supports that review and does not replace it.

What the record shows

An AI agent's likelihood of leaking data is a measured quantity, and the measurements agree: roughly a quarter to over half of tested cases in the inadvertent setting across several research groups and model families, with privacy instructions applied, and up to 70 percent under a targeted attack that has since been observed in the wild. The rate comes from helpfulness acting on a context that holds more than the task needs, which is why instructions reduce it only at the margin. The control that changes the figure is not a better prompt but a smaller context: sensitive fields that arrive as tokens cannot leak whatever the model does, which is what RedactSure's render-layer design provides. The test is to name one agent and one task, seed its context with fields the task does not need, run it repeatedly with and without an injected instruction, and count.

Frequently asked questions

Are the benchmark rates relevant to enterprise agents, or only to research settings?

The benchmarks use realistic tasks, such as drafting emails and completing web forms, with the kinds of sensitive detail enterprise records contain, so the mechanism transfers directly. The exact rate for a given enterprise agent will differ with the model, the task and the records, which is why the page recommends measuring it. What transfers without qualification is the finding that the rate is high enough to plan for and that instructions alone do not bring it down.

Do newer models leak less?

Possibly at the margin, and the benchmarks are public so a buyer can ask for current results. The structural cause, a helpful model acting on a context that holds unnecessary sensitive information, is not a property of any one model version, and the contextual integrity and PrivacyLens results span models from different vendors and years with similar findings. A plan that depends on the next model version being careful enough is a plan without a control.

What counts as a leak in these studies?

Each benchmark defines it slightly differently, which is why the figures are a range. The contextual integrity work counts disclosures that violate the norms of the context the information came from. PrivacyLens counts task outputs that include seeded sensitive details the task did not need. AgentDAM counts unnecessary use of sensitive data in completing web tasks. EIA counts successful extraction of specific personal information by an attacker. Together they cover disclosure, misuse and theft.

How does the adversarial rate combine with the inadvertent one?

They are layered, not alternative. On a day with no attacker, the inadvertent rate applies to every decision touching sensitive data. On a day when an instruction has been planted in content the agent reads, the adversarial rate applies on top. An organization's exposure is the sum over the agent's workload, and planning only for the inadvertent case assumes the content the agent reads is always benign, which for any agent reading email, documents or web pages is not a safe assumption.

Can output filtering reduce the rate?

It can reduce the rate for values that match patterns the filter knows, such as formatted account numbers. It misses values expressed in prose, names the filter has not seen and anything in a free-text field, and it operates after the model has already held the value, so it does nothing for the adversarial case where the exfiltration channel is a form or a message the filter does not inspect. Removing the value before the model reads it is the control that does not depend on the filter's coverage.

What single measurement should a risk committee ask for?

For one named agent and one named task: the number of runs, the sensitive fields seeded into context, the count of outputs containing any of them without an attacker, the same count with an instruction planted in content the agent reads, and both counts repeated with the sensitive fields tokenized before the model reads them. Five numbers from a real workflow are worth more than any published rate, and a vendor that has built for this can produce them from a pilot.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.