Skip to content
redactsure
Book a demo

Explore.

Data Report · By Chris Sowa · Published
Last updated

What Are the Risks When an AI Agent Uses Customer Information to Do Its Job?

The risk is rarely the task itself; it is the side channels, where an agent reuses customer details in places the task never required and where a hostile page can coax those details out of it. Check what customer data the agent carries beyond the step that needs it, and what duties follow if that data leaves.

Customer records flow through a central processing ring toward several separate workflow destinations.

The risk is rarely the task itself; it is the side channels, where an agent reuses customer details in places the task never required and where a hostile page can coax those details out of it. Check what customer data the agent carries beyond the step that needs it, and what duties follow if that data leaves. An agent that updates a customer's shipping address or checks an account balance is doing something a person does every day, and the task itself is not where the exposure lives. The exposure lives in everything the agent does with the information around the task: the details it holds in context after the step that needed them, the fields it fills with information that was not asked for, the pages it reads that can instruct it, and the destinations it can reach once it holds a real value. Research on web agents has measured each of these behaviors, and regulators have already written the obligations that attach when customer information leaves a firm without authorization. This page lays out what an agent typically holds when it works on a customer account, the measured ways that information leaks, how an attacker can turn an agent into a courier, which rules apply when it happens, how agents that reach customer data should be scoped, and what a buyer should refuse to accept as reassurance.

Key findings

  • Web agents built on GPT-4, Llama-3 and Claude are prone to inadvertent use of unnecessary sensitive information, meaning they carry and reuse details the task did not require (Zharmagambetov et al., 2025).
  • Even when prompted to respect privacy, GPT-4 and Llama-3-70B agents leaked sensitive information in 25.68 percent and 38.69 percent of cases in the PrivacyLens benchmark, so instructions alone do not close the channel (Shao et al., 2024).
  • Hidden page elements can steer a web agent into disclosing user data, with the Environmental Injection Attack reaching up to 70 percent success in stealing specific personal information (Liao et al., 2025).
  • The FTC Safeguards Rule requires covered financial institutions to maintain a written information security program for customer information, which includes any process by which an agent handles that information (FTC, 16 CFR Part 314).
  • Amended Regulation S-P requires covered firms to maintain an incident response program for unauthorized access to customer information and generally to notify affected individuals within 30 days (SEC, 2024).
  • An agent that nobody in security registered can reach the same customer records as a sanctioned one, which puts discovery ahead of policy in the order of work (What Is a Shadow AI Agent and How Can It Access Customer Data Without Security Knowing?).

What does an agent hold when it works on a customer account?

Take a customer account screen in a support or billing application as the worked example. It shows a name, an email address, a phone number, a billing address, the last four digits of a payment card, an account number, a plan tier, an open-ticket count and a notes field with the history of previous contacts. An agent asked to apply a promotional credit needs the account number to find the record, the plan tier to confirm eligibility, and a field to enter the credit. It does not need the phone number, the billing address, the card digits or the contact notes. All of them are on the screen, and all of them enter the model's context the moment the agent reads it.

What an agent holds, then, is not what the task needs but what the screen shows, and context does not expire at the end of a step. The details persist for the run and are available to every later decision the agent makes, including decisions about what to type into a free-text field, what to include in a summary, and what to send when a page asks for information. The AgentDAM evaluation measured this directly and found that web agents on several leading models were prone to inadvertent use of unnecessary sensitive information, using details that were present but irrelevant to the task (Zharmagambetov et al., 2025).

The permissions question compounds this. An agent operating under an employee's credentials holds whatever that employee can see, and an agent nobody registered with security holds it without anyone knowing. The problem of shadow agents reaching customer data is therefore an inventory problem before it is a policy problem: an organization cannot scope what it has not found.

How does customer information leak from an agent without an attacker?

The first class of leak needs no adversary. The agent, holding more than the task requires, uses some of it where it should not. It pastes a customer's phone number into a ticket summary that will be read by a wider team. It fills an optional form field with a billing address because the field was there and the address was in context. It includes contact notes in a message to a third party because the notes seemed relevant. None of these is an attack; each is the model exercising judgment about relevance that a person with the same information might also get wrong, and that the model gets wrong at a measurable rate.

The PrivacyLens benchmark quantified the rate. Agents built on GPT-4 and Llama-3-70B, even when given prompts designed to enhance privacy, leaked sensitive information in 25.68 percent and 38.69 percent of cases respectively (Shao et al., 2024). The finding that matters for a buyer is not the exact figure but the fact that privacy-enhancing instructions reduced the leaks without closing them. Telling the model to be careful is a control with a documented failure rate.

OWASP classifies this as LLM02:2025, Sensitive Information Disclosure, the risk that a model reveals confidential data through its outputs, and lists data sanitization before the model sees the data among the mitigations (OWASP, 2025). The ordering in that mitigation is the point. A control that acts on what the model can see prevents the leak; a control that acts on what the model says detects it, after the model has already formed the output. The behavior is the same one that justifies treating AI models like insider risks, with controls outside the model: a trusted actor with broad access and imperfect judgment is governed by what it is given, not by what it is told.

How can an attacker turn an agent into a courier for customer data?

The second class of leak does involve an adversary, and the adversary does not need access to the agent or its credentials. An agent that reads web pages, documents or emails as part of its work treats the content it reads as information, and content can contain instructions. A page the agent visits can carry text, visible or hidden, that tells the agent to enter the customer details it holds into a form on that page or to send them somewhere. If the agent complies, the attacker has used the agent's legitimate access and the agent's legitimate reading of a page to exfiltrate data the attacker never had.

The Environmental Injection Attack study tested this against generalist web agents and found that hidden page elements could steal user data with up to 70 percent attack success in extracting specific personal information (Liao et al., 2025). The mechanism is important for scoping the defense. The agent did nothing outside its permissions. It read a page it was allowed to read and typed into a field it was allowed to type into. Permission controls were satisfied throughout.

The consequence for customer information is that the risk scales with two things: how much real customer data the agent holds in context, and how many destinations it can reach while holding it. An agent that holds only the account number and plan tier can be tricked into disclosing only those. An agent that holds the whole screen can be tricked into disclosing the whole screen. Reducing what the agent holds does not stop the injection, but it bounds what the injection can take. Older systems make this worse, because legacy applications often render everything about a customer on one dense screen with no field-level access control to narrow it.

Which rules apply when an agent discloses customer information?

Obligation What it requires How an agent engages it
FTC Safeguards Rule, 16 CFR Part 314 A written information security program for customer information at covered financial institutions Any agent process that reads or moves customer information falls inside the program's scope
SEC Regulation S-P, as amended 2024 An incident response program for unauthorized access to customer information; notification to affected individuals generally within 30 days An agent disclosure to an unauthorized destination is an incident the program must detect, assess and, where required, notify
OWASP LLM02:2025 Industry guidance on preventing sensitive information disclosure by model outputs Frames the control as sanitization before the model, not inspection after

The Safeguards Rule applies to non-bank financial institutions under FTC jurisdiction and requires a written program of administrative, technical and physical safeguards for customer information (FTC, 16 CFR Part 314); an agent is a technical process handling that information and belongs in the program's risk assessment and monitoring. The SEC's 2024 amendments to Regulation S-P require covered firms to maintain an incident response program for unauthorized access to customer information and generally to notify affected individuals within 30 days (SEC, 2024). An agent induced to send customer details to a hostile page has caused that kind of unauthorized access, and the clock does not distinguish a person from a process as the cause.

IBM's annual Cost of a Data Breach report tracks the financial consequences of breaches across industries and causes and is the common reference for sizing the exposure a disclosure represents (IBM, 2025). The obligations and the costs attach to the disclosure, whoever or whatever caused it.

How should an agent that uses customer data be scoped?

Scope the data, the destinations and the actions separately, because they fail separately. On data, the discipline is to decide per field what the agent reads as content and what it merely carries as a key. The account number is a key; the agent needs to recognize the record, not to know the number. The plan tier is content; the agent must read it. The phone number, address and card digits are neither for a credit task and should not reach the model at all. Substituting the keys with consistent tokens and withholding the irrelevant fields reduces what the agent holds to what the task uses, which bounds both the inadvertent reuse that AgentDAM measured and the exfiltration that EIA measured.

On destinations, the discipline is to name where real customer values may go. For a credit task the answer is one field on one screen. Everywhere else the agent should be working with tokens, so that an instruction on a hostile page to type the customer's details into a form yields a token, not a value. On actions, the discipline is approval: every submission, payment and record change is confirmed by a named person before it executes, so an agent that has been steered toward a consequential step cannot complete it alone.

Data scope bounds inadvertent reuse, destination scope bounds exfiltration, and action scope bounds the damage from both. A platform agent that reaches customer records raises the same three questions, and the privacy questions around Meta Business Agent are a case study in asking them. The scopes should be set before the agent runs and recorded, so that the Safeguards Rule program has something to point to and the incident response program has something to compare against.

What does a governed environment change?

In a governed environment, the side channels close at the source, because an agent that holds tokens in place of a customer's phone number and account number cannot reuse them in a ticket summary or surrender them to a hostile form.

The mechanism is a class of control that sits between the rendered screen and the model. Customer values on the page beyond the fields a step reads are replaced with consistent tokens before any model reads them, so the task runs on tokens. Real values resolve only at an approved destination at the moment of an approved action, which for most customer tasks is a short list of fields in the system of record. A named person confirms what is tokenized and approves consequential actions, including payments, submissions and record changes. Every screen as the model received it, in token form, every action and every approval is recorded and exported to the organization's own logging.

This is what the three requests at the end of the buyer's list describe, and it answers the research the page rests on. Agents under privacy-enhancing prompts still leaked sensitive information in 25.68 percent and 38.69 percent of cases (Shao et al., 2024), and hidden page elements steered agents into disclosing specific personal information with up to 70 percent success (Liao et al., 2025); both results measure what an agent does with values it holds, and neither applies to values it never received. The exported record is the form an incident response program can consume, which matters where Regulation S-P sets a 30-day notification clock for unauthorized access to customer information.

The control does not change the application, the employee's permissions or the model; permission scope still sets the ceiling, and the environment changes what the agent actually holds.

Where does RedactSure sit?

The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company. Its design bounds the side channels at the point where they open. The agent runs inside a governed environment in the customer's cloud, and substitution happens at the render layer, before any model reads the screen, so the phone number, address and card digits on a customer account screen reach the model as tokens or not at all, while keys such as ACCT_001 stay consistent for the whole run. Real values resolve only at approved destinations at the moment of an approved action, which for a credit task is one field on one screen. A named person confirms the tokenization policy and approves every payment, submission and record change under Supervised Delegation. Every screen as tokens, every action and every approval lands in the AI Control Record and exports to the customer's SIEM, which gives a Safeguards Rule program and a Regulation S-P incident response program a record to work from. The design is model agnostic, needs no per-application integration, and leaves user permissions unchanged.

Deployments are in pilot. A pilot on the reader's own customer workflow would show which fields the agent ever needed in the clear, and what a hostile page could have taken. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

This page rests on peer-reviewed research and preprints that measure privacy leakage by web agents, including the AgentDAM and PrivacyLens benchmarks and the Environmental Injection Attack study, on the text of the FTC Safeguards Rule at 16 CFR Part 314 and the SEC's 2024 press release on the Regulation S-P amendments, on OWASP's LLM02:2025 entry, on IBM's Cost of a Data Breach report as an industry reference, and on earlier RedactSure research pages on customer data, shadow agents, insider-risk framing and legacy applications. All sources were read as of October 16, 2026. Quantitative figures are the cited papers' own reported results and are not extrapolated to other models or settings.

No hands-on testing of any agent, model or product was performed for this page. The leak rates and attack success rates cited were measured in the benchmarks' own environments and may not transfer to a given organization's applications. Where a vendor or platform capability is mentioned, the description is confined to the pages reviewed; what is not described there should be read as not described, not as absent. Vendor features change, and the page carries its date. Statements about RedactSure describe its current design, with deployments in pilot. The customer account screen and credit task used as examples describe a record type, not any organization; no customer or prospect is described.

Whether a given agent deployment falls within the Safeguards Rule or Regulation S-P, whether a given disclosure meets the threshold for notification, and whether a given control satisfies either program are determinations for the organization's counsel, its assessors and its own security and compliance officers, not for a research page.

What the record shows

The risk of letting an agent use customer information is concentrated in what the agent holds beyond the task and where it can send it. The benchmarks are consistent: agents reuse details the task did not need, leak at double-digit rates even when instructed not to, and can be steered by hidden page content into disclosing what they carry. The regulatory record is equally consistent: the Safeguards Rule puts the agent inside the information security program, and Regulation S-P puts an agent-caused disclosure inside the incident response program with a 30-day notification expectation. The controls that address this act before the model reads, by narrowing what it sees, and before the action executes, by requiring a person; RedactSure's design places both at the render layer. The test is to name one agent and one task and ask which fields on the screen the task reads, which it merely carries, and where a real value is ever required to appear.

Frequently asked questions

Is the risk that the agent will do the task wrong?

Usually not. Agents perform routine customer tasks such as applying a credit or updating an address with reasonable reliability, and errors in the task are caught by the same checks that catch human errors. The risk this page describes is different: the agent holds more customer information than the task needs, reuses it where it should not, and can be induced by content it reads to send it elsewhere. Those behaviors do not show up as task failures, which is why they go unnoticed.

Does telling the agent to protect customer privacy solve the problem?

It reduces leaks without eliminating them. The PrivacyLens benchmark gave agents prompts designed to enhance privacy and still measured leakage in 25.68 percent of cases for GPT-4 and 38.69 percent for Llama-3-70B. An instruction is a behavioral request to a system with imperfect judgment, and it has a published failure rate. Controls that decide what the model can see, and that require a person to approve consequential actions, do not depend on the model's compliance.

How can an attacker reach customer data through an agent they do not control?

By placing instructions in content the agent reads. A web page, document or email can contain text, sometimes hidden, that tells the agent to enter the information it holds into a form or send it to an address. The Environmental Injection Attack study measured up to 70 percent success in stealing specific personal information this way. The agent stays within its permissions throughout; the attacker borrows its access and its context rather than breaking either.

Which regulations apply when an agent discloses customer information?

For non-bank financial institutions under FTC jurisdiction, the Safeguards Rule at 16 CFR Part 314 requires a written information security program that an agent process falls inside. For SEC-regulated broker-dealers, investment companies and advisers, amended Regulation S-P requires an incident response program and generally notification of affected individuals within 30 days. Other sectors carry their own rules. In each case the obligation attaches to the disclosure regardless of whether a person or a process caused it.

Is monitoring the agent's output enough to catch disclosures?

No. Output monitoring sees what the model composes after it has composed it, and a disclosure typed into a field on a hostile page may bypass the monitored channel entirely. Inadvertent reuse, such as a phone number pasted into a ticket note, generates no alert at all. OWASP places data sanitization before the model among the mitigations for sensitive information disclosure because preventing the model from holding the data is the control that works regardless of output path.

What is the single most useful question to ask a vendor?

Ask for a field-level statement of what the agent sees on each screen of one real workflow, with the fields the task does not need shown as withheld or tokenized. The answer reveals whether the vendor's control acts on what the model receives or only on what it produces. A vendor who can show the screen as the model sees it, with irrelevant customer fields absent, has bounded the side channels. A vendor who describes the model's instructions has not.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.