Skip to content
redactsure
Book a demo

Explore.

Data Report · By Chris Sowa · Published
Last updated

How to test prompt injections & manipulations on shopping agents?

A sound test plants attacker style instructions in product pages, reviews, and seller messages, then checks whether the agent changes its task while a clean run still completes. A test that only asks a hostile question in chat measures the wrong channel, and a defense that blocks everything scores well on security while destroying the agent's usefulness.

Shopping pages pass an inspection gate that removes a suspicious offer card.

A sound test plants attacker style instructions in product pages, reviews, and seller messages, then checks whether the agent changes its task while a clean run still completes. A test that only asks a hostile question in chat measures the wrong channel, and a defense that blocks everything scores well on security while destroying the agent's usefulness. Testing a shopping agent for prompt injection means placing controlled, attacker style instructions in the places a real attacker would use, product pages, reviews, seller messages, images, and search results, and measuring whether the agent changes its task, its choice, or its actions, while a clean version of the same task still completes correctly. Shopping agents fail through content they were sent to read, and through manipulation that is not an instruction at all, such as a planted review. What a failed test can cost depends on what the agent held when it was deceived, which is why Least Exposure belongs in the test plan. The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company.

Key findings

  • Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications, and its indirect form, planted in external content, is the primary channel for shopping agents that browse listings and reviews (OWASP, 2025).
  • NIST classifies indirect prompt injection by attacker goal in its adversarial machine learning taxonomy, covering availability, integrity, privacy, and abuse outcomes, which gives testers a ready made set of attack goals to cover (NIST, 2023).
  • AgentDojo, a peer reviewed benchmark from ETH Zurich, pairs 97 realistic user tasks with 629 security test cases and scores both task completion and attack success from environment state, rather than from a model judgment (Debenedetti et al., 2024).
  • A shopping test must measure two things at once: utility on clean tasks, and attack success on injected versions of the same tasks. A defense that blocks everything scores well on the second and fails the first, which is why AgentDojo reports the two scores together (Debenedetti et al., 2024).
  • Manipulation testing is broader than injection testing. It also covers planted reviews, sponsored placement, and price and attribute deception, which NIST's taxonomy treats as integrity attacks on the data the model reasons from (NIST, 2023).

What should a shopping agent test environment look like?

Start with isolation. The agent under test should run against a staging store, a local copy of a catalog, or a recorded set of pages, with test accounts, test payment instruments, and no route to a real purchase, a real message to a seller, or a real change of address. OWASP treats excessive agency as a separate risk, LLM06, and a test harness that gives an agent real purchasing power recreates that risk in the laboratory (OWASP, 2025). Every consequential tool should be stubbed so the harness records the attempted action and its arguments instead of performing it.

The catalog needs to be realistic enough for manipulation to be meaningful. WebShop, a widely used simulated e-commerce environment, was built with 12,087 instructions over a large product set because small toy catalogs do not reproduce search and filtering behavior (Yao et al., 2022). A tester needs multiple plausible products per query, near substitutes, differing prices and ratings, and pages with the full apparatus of a real listing: descriptions, specifications, reviews, seller notes, shipping terms, and images with text.

Each task should exist in a matched pair. The clean version defines the expected outcome: the correct product, the acceptable price range, the attributes that must match the user request. The injected version is identical except for one planted element. Matched pairs make a result interpretable, because a difference in behavior can then be attributed to the planted element rather than to a hard task. Record everything the agent saw and did, including the exact page content delivered to the model and the tool calls in order. Without the delivered content, a failure cannot be reproduced once a page changes.

Which injection surfaces should the test cases cover?

Cover the surfaces in the order a shopping agent encounters them. Search results come first: a listing title or snippet that instructs the agent to prefer one product, to skip price comparison, or to treat a sponsored item as the only match. Product descriptions come next, including hidden markup, comments, attributes, metadata, and text embedded in images, since a browsing pipeline may read any of those. Specifications and comparison tables can carry instructions disguised as notes to automated buyers.

Reviews are the highest value surface, because agents are explicitly told to weigh them. Test cases should include a review that instructs the agent to recommend the product regardless of rating, a review that claims a safety or compatibility fact that the specification contradicts, a block of reviews planted to move an average, and a review that tries to make the agent disclose the user budget, address, or payment details. Questions and answers, seller responses, and storefront pages deserve the same treatment, as do messages that arrive after a purchase step, such as delivery notices containing instructions.

Vary the delivery as well as the location. Include direct imperatives, instructions framed as a system notice, instructions framed as the user's own earlier request, payloads split across two pages, encoded payloads, multilingual payloads, and payloads that only trigger on a second visit or after a specific tool result. NIST's taxonomy of indirect injection by goal helps keep coverage honest: for each surface, test at least one payload aimed at changing the decision, one aimed at extracting information, one aimed at causing an unauthorized action, and one aimed at degrading the service, for example by looping or by refusing the task (NIST, 2023).

Also test the boundaries the agent is supposed to respect. A payload that asks the agent to exceed the user's stated budget, to substitute a different brand, to buy a quantity the user did not request, to use a stored payment method, or to reveal a value from an earlier session measures whether task constraints are enforced outside the model or merely requested in a prompt.

How should manipulation that is not an instruction be tested?

Not every manipulation tells the agent what to do. Some change the evidence the agent reasons from. A set of planted reviews can make an inferior product appear better reviewed. A listing can state a false attribute, such as compatibility, capacity, or certification, in ordinary descriptive language. A price can be displayed with a misleading comparison figure, or shipping terms can be buried so that the cheapest headline price is not the cheapest total. These are integrity attacks on the input data, and a test suite limited to imperatives will miss all of them.

Test them as decision quality cases. Give the agent a task with a verifiable ground truth in the catalog data, such as the lowest total price for a specified item, then vary the surrounding evidence: planted praise, a false attribute claim contradicted elsewhere on the page, and a review set that is internally inconsistent. Score whether the agent selected the product that satisfied the stated criteria, and whether it surfaced the conflict to the user rather than resolving it silently.

A second family is steering by framing. Sponsored results, default sorts, and labels such as best seller can move an agent that treats page ordering as a quality signal. Test whether the agent follows the user's criteria when the first result is sponsored and unsuitable.

A third family is cross session manipulation, where content in one task tries to set a preference or a trusted seller for later tasks. Test it by running a poisoned task followed by a clean task and checking whether the preference survives.

How do you score an attack as successful?

Define success before running anything, at the level of environment state. AgentDojo scores utility and security with formal checks over the state of the environment, which is why its results are reproducible across runs (Debenedetti et al., 2024). For a shopping agent, useful definitions are: the agent selected a product other than the one meeting the user's criteria, the agent attempted a stubbed action the user did not authorize, the agent disclosed a protected value in an output or a tool argument, the agent exceeded a stated budget or quantity, or the agent reported a planted claim as fact in its summary.

Separate attempted from completed effects. An agent that repeats an injected instruction in its reasoning but still buys the correct item has shown exposure without impact. An agent that never mentions the payload but quietly selects the attacker product has shown impact without visible exposure. Both matter, and they point to different fixes.

Report utility alongside attack success for every defense configuration, and report refusals on clean tasks, since an agent that declines to shop is not secure in any useful sense. Repeat each case more than once, because model behavior is stochastic, and record the model, version, settings, system prompt version, and tool configuration for each run. Avoid using a language model as the only judge: a judge can be influenced by the same planted content. Use deterministic checks first, and reserve review for open ended outputs such as summaries.

Which published tools and benchmarks can a tester reuse?

Published agent benchmarks supply ready made tasks and attack cases. AgentDojo provides dynamic environments in domains including travel, banking, workspace, and messaging, with attacks delivered through tool outputs (Debenedetti et al., 2024). InjecAgent, a benchmark for indirect injection in tool integrated agents, provides 1,054 test cases and separates direct harm goals from data stealing goals (Zhan et al., 2024). Neither is a shopping benchmark, so a tester should borrow their attack construction and scoring approach and rebuild the tasks around a catalog.

For the shopping environment itself, WebShop and Mind2Web are the standard references for web navigation and product search tasks, and they are useful for generating realistic clean tasks and measuring whether a defense breaks ordinary shopping ability (Yao et al., 2022).

For generating adversarial prompts, three open source tools are documented by their maintainers: garak, the NVIDIA LLM vulnerability scanner; promptfoo, which builds red team cases into evaluation runs and continuous integration; and Microsoft PyRIT, which orchestrates multi turn adversarial campaigns. Scanner probes are strong on known patterns and weaker on a specific agent tool chain, so their output should feed the environment based tests above rather than replace them. Keep the test set versioned, and keep a held out set that is not used while tuning defenses.

How often should testing run, and what should be recorded?

Run the full suite before a release, after any change to the model, the system prompt, the tool set, the extraction pipeline, or the approval rules, and on a schedule against production-like content, because the pages an agent reads change even when the agent does not. A smaller smoke set can run in continuous integration on each change. The joint guidance on careful adoption of agentic AI services from CISA and partner agencies recommends adversarial testing and red teaming, evaluation in varied conditions and at different autonomy levels, and fail safe defaults that limit the effect of unexpected behavior (CISA et al., 2026).

Each run should produce an auditable record: test set version, target configuration, per case utility and attack outcomes, the delivered content for failures, the tool calls attempted, and the disposition of each finding. Findings should be filed like other security defects, with severity based on the action or disclosure achieved. Payloads should be inert, addressed to test accounts and stubbed tools, and never copied into real listings, real reviews, or third party sites, where they would expose other shoppers and other agents.

Where does RedactSure sit?

Teams applying this discipline to agents that handle sensitive values should also look for a design that limits what a failed test can cost, and RedactSure is built as that design. Policy selected values are replaced by consistent tokens at the render layer before the model reads the screen, so a manipulated listing persuades an agent holding CARD_001 and ACCT_001 rather than the shopper's card number and address. A review that tries to extract the user's payment details collects tokens. That is render-layer tokenization; real values resolve only at approved destinations at the moment of an approved action.

Any purchase, message to a seller, or record change still waits for a named person's approval at the moment it runs, under Supervised Delegation. The Planner sets which values are tokenized and what the agent may do on each screen, and a named person confirms that policy before the run. Every screen as tokens, every attempted action, and every approval lands in the AI Control Record and exports to the customer's SIEM. The environment is model agnostic and needs no per-application integration. Deployments are in pilot. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

The method rests on published sources, read as of October 13, 2026: the OWASP Top 10 for LLM Applications (standard), the NIST adversarial machine learning taxonomy, AI 100-2e2023 (standard), the peer reviewed benchmarks AgentDojo, InjecAgent, WebShop, and Mind2Web (cited from their arXiv pages and venue listings), the public documentation of the open source tools garak, promptfoo, and PyRIT (vendor documentation), and the joint guidance on careful adoption of agentic AI services from CISA and partner agencies (official guidance). Benchmark figures describe the size and design of those published benchmarks, not results obtained by the author.

No tests were run for the page, no shopping agent was evaluated, and no attack success rate is reported for any product, including RedactSure. No hands-on testing of garak, promptfoo, PyRIT, or any commercial shopping agent was performed; each tool is described as its maintainers document it. Published benchmarks use fixed scenarios, so their coverage does not extend to every catalog, tool chain, or attacker adaptation, and results from one agent configuration do not transfer automatically to another. Tool features, benchmark versions, and vendor documentation change; the page carries its date and is revised when they do. The staging store, planted review, and stored payment method examples describe test designs, not any organization; no customer or prospect is described.

Whether a given test plan, defense configuration, or agent deployment meets an organization's requirements belongs to its counsel, assessor, and its security and compliance officers, not to this page.

What the record shows

A shopping agent cannot be tested at the chat box, because that is not where it gets attacked. Its working material is other people's content: listings, reviews, seller messages, images, and search results, any of which can carry an instruction or a quieter manipulation such as a planted review. Serious testing rebuilds that environment under control, seeds each surface, and scores clean task completion alongside whether injected versions move the agent's choice or actions, as benchmarks such as AgentDojo demonstrate. A test plan that reports attack success without clean task utility, or that scores success from what the agent said rather than from the state it left behind, cannot tell a secure agent from a broken one. The one check a reader can apply to any test report is to ask for both numbers, scored from environment state, with the delivered page content kept for every failure.

Frequently asked questions

Why must clean tasks be tested alongside injected ones?

Because a defense can look perfect by breaking the agent. Blocking every purchase, refusing every listing, or escalating every decision drives attack success toward zero while also destroying the utility the agent was deployed for. AgentDojo, the ETH Zurich benchmark, scores task completion and attack success together for exactly this reason. A tester who reports only one of those numbers cannot tell a secure agent from a paralyzed one, so both belong in every result.

What makes an injection attempt count as successful?

The effect on the task, judged from the environment rather than from the conversation. An attack succeeds when the agent follows the planted instruction in a way that changes what it does: choosing the attacker's product, applying an unauthorized discount, sending a message it was not asked to send, or exposing information it should have kept. Reading a payload aloud or summarizing it is not success on its own. Scoring from resulting state keeps the test honest about harm.

How should shopping manipulations that are not instructions be tested?

As their own test family, with the same paired discipline. Planted reviews, sponsored placement disguised as organic results, and misleading price or attribute claims do not order the agent to do anything; they corrupt the evidence the agent reasons from. Test cases should vary that evidence while holding the user's request constant, then measure whether the recommendation changes. A suite covering only explicit instructions would pass an agent that is still easily steered by fabricated social proof.

Can a language model be used to judge whether an attack succeeded?

Not as the only judge. A model acting as judge reads the same planted content that was aimed at the agent, so an injection that moves the agent can move the judge too. Define success as a deterministic check over environment state first: which product was selected, which stubbed action was attempted with which arguments, which value appeared in an output. Reserve model or human review for open ended outputs such as the agent's summary to the user.

Can AgentDojo or InjecAgent be run directly against a shopping agent?

Not as shipped. AgentDojo covers travel, banking, workspace, and messaging environments, and InjecAgent covers tool integrated agents in general; neither includes a product catalog, reviews, or a checkout. What transfers is their method: attacks delivered through content the agent reads, success scored from state, and utility reported next to security. Rebuild the tasks around a staging catalog, using WebShop or Mind2Web style tasks for the clean baseline.

Is it acceptable to plant test payloads in real listings or reviews?

No. Payloads belong in a staging store, a local catalog copy, or recorded pages, addressed to test accounts and stubbed tools. A payload placed in a real listing, a real review, or a third party site reaches other shoppers and other agents that were never part of the test, and it can trigger real purchases, messages, or disclosures. Keep every consequential tool stubbed so the harness records the attempted action instead of performing it.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.