Skip to content
redactsure
Book a demo

Explore.

Data Report · By Chris Sowa · Published
Last updated

Is It Safe to Let Enterprise AI Agents Access Personally Identifiable Information?

Not if safety depends on the model deciding when personal information is appropriate to share, because research measures that judgment failing in a third to a half of cases. Check whether your controls act on what the agent receives, which is a design property, or on what it chooses to disclose, which is a behavior.

Many personal record cards converge through a narrow opening and emerge as a smaller set of task records.

Not if safety depends on the model deciding when personal information is appropriate to share, because research measures that judgment failing in a third to a half of cases. Check whether your controls act on what the agent receives, which is a design property, or on what it chooses to disclose, which is a behavior. The question is usually asked as if it had a yes or no answer, and it does not. An agent that reads a policy record to renew a policy is accessing personally identifiable information, and the renewal will happen either way. What varies is whether the agent's access is bounded by design or by the model's own sense of what is appropriate. Three independent studies have now measured that sense, in conversation and in agent action, and found it unreliable in a way that instruction does not repair. The regulatory framework for personal information, meanwhile, has always located protection in controls rather than in the discretion of whoever holds the data. This page sets out what an agent receives when it reads a record carrying personal information, what the research measures about model judgment, why instructions reduce but do not close the gap, how established guidance frames the control, what a design that bounds access looks like, and how a buyer should test an agent before extending it to records that carry the data an enterprise agent can actually reach.

Key findings

  • GPT-4 and ChatGPT reveal private information in contexts where humans would not in 39 percent and 57 percent of cases respectively, when tested against contextual integrity norms (Mireshghallah et al., 2024).
  • Even with privacy-enhancing prompts, GPT-4 and Llama-3-70B agents leak sensitive information in 25.68 percent and 38.69 percent of cases, so instruction narrows the failure without removing it (Shao et al., 2024).
  • Web agents built on GPT-4, Llama-3 and Claude are prone to inadvertent use of unnecessary sensitive information, using details that were present in their context but not required by the task (Zharmagambetov et al., 2025).
  • NIST's guide to protecting the confidentiality of personally identifiable information states that PII should be protected from inappropriate access, use and disclosure, and locates that protection in organizational and technical controls (NIST, 2010).
  • OWASP lists sensitive information disclosure as LLM02:2025 and places data sanitization before the model among its mitigations, which is a control on what the model receives rather than on what it says (OWASP, 2025).
  • An agent whose access was never registered with security reaches the same personal information as a sanctioned one, which makes inventory a precondition for any access decision (What Is a Shadow AI Agent and How Can It Access Customer Data Without Security Knowing?).

What does an agent receive when it reads a record with personal information?

Take an insurance policy record as the worked example. The screen shows the policyholder's name, date of birth, home address, phone number, email, driver's license number, the vehicle identification numbers, the premium, the coverage limits and the renewal date. An agent asked to prepare the renewal needs the renewal date, the premium, the coverage limits and a key to confirm it is working on the right policy. It does not need the license number, the birth date, the address or the phone number for that step. An agent that works through the screen receives all of them, because the screen is what it perceives, and every value on it enters the model's context.

Context persists for the run. The license number the agent did not need for the renewal is still in context when the agent writes the renewal confirmation, fills a form on the insurer's partner portal, or summarizes the account for a colleague. At each of those points the model decides, implicitly, whether to include it. That decision is the one the research has measured, and it is the decision this page is about.

The permissions model does not help here. An agent operating under an underwriter's credentials sees what the underwriter sees, which is sized for the underwriter's whole job rather than for a single step. The question of how much enterprise data an agent can actually reach has a large answer in most organizations, and the answer is the same for a registered agent and for one nobody in security knows about. Access is the ceiling of exposure. The probability of exposure depends on what the model does with what it holds, which is where the studies come in.

What does the research measure about model judgment on personal information?

Three studies measure the same thing from different angles: whether a language model, holding personal information, shares it in contexts where a person would not. The first tested models against contextual integrity theory, the framework in which the appropriateness of an information flow depends on the context in which it was shared. GPT-4 and ChatGPT revealed private information in contexts that humans would not in 39 percent and 57 percent of cases (Mireshghallah et al., 2024). These were conversational settings, and the failure was one of judgment rather than of capability: the models understood the information, and shared it anyway.

The second moved the test to agents acting in realistic tasks. PrivacyLens evaluated privacy norm awareness in action and found that GPT-4 and Llama-3-70B agents leaked sensitive information in 25.68 percent and 38.69 percent of cases even when given prompts designed to enhance privacy (Shao et al., 2024). The third tested web agents performing tasks in a browser. AgentDAM found that agents built on GPT-4, Llama-3 and Claude were prone to inadvertent use of unnecessary sensitive information, drawing on details in context that the task did not require (Zharmagambetov et al., 2025).

The convergence matters more than any single number. Across three research groups, three evaluation designs and several model families, the finding is that a model holding personal information cannot be relied on to withhold it when the context calls for withholding. The rates vary with the model and the test, and they fall with better prompting, but they do not approach zero in any study reviewed. For an enterprise deciding whether an agent may access personal information, the finding reframes the decision: the agent's judgment is not the control, so something else must be.

Why do instructions reduce the problem without closing it?

An instruction to protect privacy is a request to a probabilistic system to weigh one consideration among many at every step. The system weighs it, and the research shows the weight is real: PrivacyLens measured lower leak rates with privacy-enhancing prompts than without. But the weight competes with the instruction to complete the task, with the content the agent reads during the task, and with the model's own patterns about what is relevant. At a measurable fraction of decision points the other considerations win.

Content the agent reads is the sharpest competitor. A page, a document or a form the agent encounters can ask for information in a way that looks like part of the task. An instruction to protect privacy must be applied by the model against that request, in the moment, and the studies show it loses a quarter to a third of the time with privacy prompting in place (Shao et al., 2024). The attacker, if there is one, does not need to defeat the instruction. The attacker needs to be in the fraction where it does not hold.

This is why the established framing for personal information never relied on the holder's discretion. NIST's guide to protecting the confidentiality of PII states that PII should be protected from inappropriate access, use and disclosure, and sets out controls for doing so rather than guidance on judgment (NIST, 2010). The approach that treats AI models like insider risks, with controls outside the model follows the same logic: a trusted actor with imperfect judgment is governed by what it is given and what it is permitted to do, not by what it is asked to remember.

How does established guidance frame the control?

Framework What it asks for How it applies to an agent reading personal information
NIST SP 800-122 Protect PII from inappropriate access, use and disclosure The agent's view is an access; its outputs are uses and disclosures
NIST SP 800-53 Rev. 5 Controls including least privilege and access enforcement A whole screen for a five-field task is not least privilege
NIST Privacy Framework 1.0 Deliberate data processing decisions Which fields the agent receives is decided in advance
OWASP LLM02:2025 Sanitize data before the model Control what the model receives, not only what it produces

NIST SP 800-122 defines the problem in terms of access, use and disclosure and treats each as something to be controlled (NIST, 2010). SP 800-53 Revision 5 supplies the control catalog, including least privilege and access enforcement, and an agent that receives every field on a screen to perform a task that reads five of them is outside least privilege by construction (NIST, 2020). The Privacy Framework asks organizations to make data processing decisions deliberately as part of enterprise risk management, and which fields an agent receives is such a decision, to be made before the run rather than improvised by the model during it (NIST, 2020).

OWASP's LLM02:2025 entry brings this to the model: among its mitigations for sensitive information disclosure it lists sanitizing data before it reaches the model, the one mitigation that does not depend on the model's behavior (OWASP, 2025). The frameworks agree on the location of the control: before the model, not after.

What does an access design that does not rely on model judgment look like?

Return to the policy record. A design that bounds access decides, per field and before the run, what the model receives. The renewal date, premium and coverage limits reach the model as readable values because the task reads them. The policyholder's name and license number reach the model as consistent tokens, such as ACCT_001, because the task needs them only as keys to recognize the same policy on the next screen. The birth date, address and phone number do not reach the model at all, or reach it as tokens, because the renewal step does not use them. The model's judgment about whether to disclose the license number is no longer load-bearing, because the model does not hold the license number.

Where the task finally requires a real value, such as the policyholder's name on the renewal notice, the value is restored at that destination at the moment the approved action is taken. The decision about where real values may appear is made in advance and recorded. Consequential actions, such as sending the notice or changing the record, are approved by a named person before they execute.

The design has to live where the agent perceives, since an agent that reads the policy system, the payments console and the partner portal through a screen passes through all three. Tools that classify files and permissions at rest see which records contain personal information but not which fields a given agent received on a given screen, a distinction examined in the comparison of a render-layer design with Varonis. The audit question follows: the record of the run should show what the agent accessed across applications as the model saw it, so that the access decision can be checked after the fact.

What does a governed environment change?

In a governed environment, the safety of letting an agent reach personally identifiable information stops depending on the model's judgment about when to disclose it, because the identifiers the task does not need are not in the model's context to be disclosed.

The mechanism is a class of control that sits between the rendered screen and the model. Personal identifiers on the page beyond the fields a step reads are replaced with consistent tokens before any model reads them, so the task runs on tokens. Real values resolve only at an approved destination at the moment of an approved action. A named person confirms what is tokenized and approves consequential actions, including payments, submissions and record changes. Every screen as the model received it, in token form, every action and every approval is recorded and exported to the organization's own logging.

This is the design property the page's tests look for. The research measures disclosure as a behavior: even under privacy-enhancing prompts, GPT-4 and Llama-3-70B agents leaked sensitive information in 25.68 percent and 38.69 percent of cases (Shao et al., 2024), so an agent holding real values will supply them at some rate when a form asks. An agent holding tokens can supply only tokens, and repeated runs do not change that. The record also answers the third test: a log that shows each screen as the model received it, not only the actions taken, is what is needed to say whether PII was accessed inappropriately in the sense SP 800-122 uses (NIST, 2010).

The control does not change the application, the user's permissions or the model; the model is no more careful than before, and the environment changes only which identifiers are on the screen when it looks.

Where does RedactSure sit?

The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company. Its design removes the model's judgment from the path by deciding what the model receives. The agent runs inside a governed environment in the customer's cloud, and substitution happens at the render layer, before any model reads the screen, so the license number, birth date and contact details on a policy record reach the model as consistent tokens such as ACCT_001 for the whole run, or not at all, while the renewal date and premium the task reads stay readable. Real values resolve only at approved destinations at the moment of an approved action. A named person confirms the tokenization policy and approves every payment, submission and record change under Supervised Delegation. Every screen as tokens, every action and every approval lands in the AI Control Record and exports to the customer's SIEM, which gives the access question an answer that can be checked. The design is model agnostic, needs no per-application integration, and leaves user permissions unchanged.

Deployments are in pilot. A pilot on the reader's own workflow would show each screen as the model received it and whether any personal identifier the step did not read was ever in the clear. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

This page rests on peer-reviewed research and preprints that measure language model and agent behavior with personal information, including the contextual integrity study, PrivacyLens and AgentDAM, on NIST SP 800-122, SP 800-53 Revision 5 and the Privacy Framework Version 1.0 as official guidance, on OWASP's LLM02:2025 entry, and on earlier RedactSure research pages on customer data, data access, auditing and shadow agents. All sources were read as of October 16, 2026. Quantitative figures are the cited papers' own reported results and are not extrapolated to other models, prompts or tasks.

No hands-on testing of any agent, model or product was performed for this page. The leak and disclosure rates cited were measured in the studies' own settings, and a given model in a given deployment may perform differently; the page relies on the direction and convergence of the findings rather than on any single figure. Where a vendor capability is mentioned, the description is confined to the pages reviewed; what is not described there should be read as not described, not as absent. Vendor features change, and the page carries its date. Statements about RedactSure describe its current design, with deployments in pilot. The insurance policy record and renewal task used as examples describe a record type, not any organization; no customer or prospect is described.

Whether a given agent's access to personal information meets an organization's obligations under privacy law, sector regulation or its own policies is a determination for its counsel, its assessors and its own security and privacy officers, not for a research page.

What the record shows

Letting an enterprise agent access personal information is safe to the extent that safety does not depend on the model. Three independent studies measured model judgment about when personal information is appropriate to share and found it failing in roughly a quarter to a half of cases, with privacy instructions narrowing the gap but never closing it. The guidance that governs personal information has always placed protection in controls on access, use and disclosure rather than in the holder's discretion, and OWASP's placement of sanitization before the model carries that principle to agents. The design that follows decides per field what the model receives, keeps keys consistent as tokens, restores real values only at approved destinations, and records what the model saw; RedactSure's design does this at the render layer. The test is to name one agent and one task and ask which personal identifiers the step read, and which the model received anyway.

Frequently asked questions

Can a well-prompted agent be trusted with personal information?

Trusted more than an unprompted one, but not relied on. PrivacyLens measured leakage of 25.68 percent for GPT-4 agents and 38.69 percent for Llama-3-70B agents with privacy-enhancing prompts in place. A prompt is a consideration the model weighs against the task and against content it reads, and at a measurable fraction of decision points it loses. Controls that decide what the model receives do not depend on the weighing coming out right.

Is the problem that models cannot recognize personal information?

No. The contextual integrity study found that models understood the information and shared it anyway in 39 and 57 percent of cases, in contexts where people would not. The failure is of judgment about appropriateness, not of recognition. That distinction matters for the control: a classifier that recognizes personal information is not enough if the model then decides, wrongly, that sharing it fits the context. The control has to remove the decision rather than inform it.

Does an agent operating under an employee's permissions have appropriate access?

It has the employee's access, which was sized for the employee's whole job. For any single step, that access is wider than the step requires, and least privilege in NIST SP 800-53 asks for access sized to the task. The agent is not exceeding its permissions; it is exercising permissions that were never meant to bound a single operation. Narrowing what the agent receives per step is how least privilege is restored without changing the employee's own access.

What is the difference between controlling what the model receives and filtering what it outputs?

Timing and coverage. Output filtering inspects text the model has already composed, on the channels the filter watches. A disclosure typed into a field on a third-party page, or carried into a later step's reasoning, may never pass through the filter. Controlling what the model receives means the personal information is not in context to be disclosed through any channel. OWASP lists sanitization before the model among its mitigations for this reason.

Does tokenizing personal information stop the agent from doing its job?

Not when the tokens are consistent for the run. The agent uses personal identifiers mainly as keys, to recognize the same policy or person across screens, and a stable token such as ACCT_001 serves that role. Fields the task reads as content, such as a renewal date or a premium, stay readable. Where a real value is finally required, such as a name on a notice, it is restored at that destination. The task that fails is one whose policy masked content the step needed.

How should a pilot measure whether access was safe?

By capturing each screen as the model received it and comparing it with the fields the step read, by placing a plausible request for personal information in the workflow and observing what the agent supplies across several runs, and by checking that the run's record shows what the model saw rather than only what it did. Those three measurements answer the access, disclosure and audit questions that NIST SP 800-122 frames, on the buyer's own records.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.