Skip to content
redactsure
Book a demo

Explore.

Explainer · By Chris Sowa · Published
Last updated

What Is AI Agent Auditability and Can You Prove What an Agent Actually Touched?

AI agent auditability is the property that lets an independent reviewer reconstruct, from records the organization holds, what an agent was shown, what it did and who approved each consequential action, to a standard an auditor would accept as evidence. Without it, every statement about agent behavior is a vendor's assurance, and nobody can answer the question a regulator asks.

File, database and document access become task records that pass approval into an audit file rack.

AI agent auditability is the property that lets an independent reviewer reconstruct, from records the organization holds, what an agent was shown, what it did and who approved each consequential action, to a standard an auditor would accept as evidence. Without it, every statement about agent behavior is a vendor's assurance, and nobody can answer the question a regulator asks. Logging is the activity; auditability is the result. A deployment can log a great deal and still be unauditable, because the log omits what the model received, cannot be tied to a named person, or sits in a vendor's system the organization cannot inspect or preserve. The mechanism built for that purpose is the AI Control Record: the record of every screen the agent saw as tokens, every action, every approval with a name, exported to the customer's own monitoring. The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company.

Key findings

  • The federal control catalog defines what an audit record must contain: the type of event, when and where it occurred, its source, its outcome and the identity of those involved, in the AU family of NIST SP 800-53 (NIST, 2020).
  • The same catalog treats protection of audit information, non-repudiation and retention as controls separate from logging, so a log that exists is not a record that proves anything (NIST, 2020).
  • Technical verification schemes for AI claims rest on unexamined assumptions and contain systematic fallacies a dishonest developer could exploit, so a vendor's self-reported assurance produces misplaced trust rather than evidence (Paris, Moon and Guo, 2026).
  • Executives define trust in AI operationally as accuracy, predictability, consistency and traceability, and 77 percent say the true benefits of AI depend on that foundation (Accenture, 2025).
  • What reached a model and what influenced its output are separate questions, and tracking which inputs shaped an output is an open research problem once data is inside the context (Siddiqui et al., 2025).
  • Agent platforms now offer session recording with verbosity configurable from all data with screenshots down to minimal, so the content of the record is a choice the organization has to make (Microsoft, 2026).

What does auditability mean when the actor is an AI agent?

A system is auditable when the events that matter are recorded, the records can reconstruct what happened, they cannot be altered or denied afterwards, and they are kept long enough for someone to look. The AU family of NIST SP 800-53 sets this out as controls: which events to log, what each record contains, how records are reviewed and protected, how an actor is prevented from denying an action, how long records are kept and how they are generated (NIST, 2020). An AI agent operating an organization's applications is such a system.

For a human user, the events are logins, record opens, edits, submissions and payments, each attached to the user's account. For an agent, three more things have to be in the record. The first is what the agent was shown: the content of every screen, document or tool result that entered the model's context, because that content is what the agent reasoned from and what an injected instruction would have arrived in (Greshake et al., 2023). The second is what the agent did, including the drafts and navigation the destination system's log never sees. The third is who approved what, by name, at the moment a consequential action ran, because the agent's account is not a person who can be asked.

Auditability means all three can be reconstructed from records the organization controls. An audit trail is the artifact; auditability is whether the artifact, together with how it is protected and retained, would stand up when an assessor, an opposing counsel or a regulator asks what the agent touched. A deployment that logs tool calls but not screen content, or keeps everything in a vendor's console it cannot export, has an audit trail without auditability.

Why is an agent harder to audit than a user or a script?

A script does the same thing every time it runs, so its code is a sufficient record of its behavior. A user can be interviewed. An agent is neither. The model that drives it can produce different actions from the same inputs on different days, and the inputs include everything on the screens it read. The Microsoft AI Red Team's taxonomy of failure modes in agentic systems catalogs failures where behavior depends on what the agent encountered during the run rather than on a fixed program (Microsoft AI Red Team, 2025). The configuration did not determine the behavior, so the record of the run is the only evidence there is.

The second difficulty is that the agent reads far more than it uses. A computer-using agent that takes screenshots receives everything on the page, so the record of what it was shown is a record of exposure as well as of behavior. Research on information flow through language models makes the point from the other direction: which inputs influenced a given output is a separate question from which inputs were present, and tracking that influence is an open problem (Siddiqui et al., 2025). An auditor asking whether a customer's account number affected the agent's decision cannot get a clean answer from the model. The only clean answer is that the number was never in the context.

The third difficulty is attribution. The agent acts under a user's credentials or a service identity, so the destination system's log attributes the action to an account rather than to a decision. When a payment is released, the question is whether a named person approved it, and when. Non-repudiation, in the AU family's sense of a record that prevents an actor from falsely denying an action, has to extend to the approval itself (NIST, 2020).

What standard does the record have to meet to count as evidence?

The way to set the bar is to take each AU control and ask what it requires when the system is an agent.

AU control (NIST SP 800-53) What it requires in general What it requires for an AI agent
Event logging (AU-2) A defined list of events the organization logs Every screen the model received, every action, every approval, every pause and takeover
Content of audit records (AU-3) Type, time, location, source, outcome and identity for each event Two identities: the agent and the named person who approved the step
Audit record review (AU-6) Regular review of records for inappropriate activity Review of what the model was shown, not only what it submitted
Protection of audit information (AU-9) Records protected from unauthorized access, modification and deletion Records held by the customer, under the customer's keys, not only in the vendor's console
Non-repudiation (AU-10) An actor cannot falsely deny having performed an action The approval carries a name and a time that neither the person nor the agent can later dispute
Audit record retention (AU-11) Records kept for a defined period to support investigation Retention set by the organization's policy, not by a vendor default
Audit record generation (AU-12) The system generates the records for the defined events Generation inside the governed environment where the screen was rendered, not reconstructed afterwards

The AU family's definitions are the source for the general column (NIST, 2020); the agent column applies each control to the three things an agent record must hold. Two rows do most of the work. Protection of audit information fails whenever the only copy lives with the vendor that also runs the agent, because the organization cannot show it was not altered and cannot produce it if the relationship ends. Non-repudiation fails whenever approval is a setting rather than an event: an "always allow" grant made once cannot be tied to the specific payment it later authorized, the argument of the page on standing permissions.

Why is a vendor's own assurance not evidence?

The question for auditability is not whether a vendor's safety description is accurate but whether the organization can check it. Paris, Moon and Guo examine the technical verification schemes offered for AI claims, cryptographic and statistical, and find that they rest on unexamined assumptions and contain systematic fallacies a dishonest developer could exploit, so that the schemes produce misplaced trust rather than accountability (Paris, Moon and Guo, 2026). The practical reading for an enterprise is that it needs its own record of what the agent saw and did, generated where the work happened and held under its own control.

Accenture's Technology Vision 2025 frames the requirement from the buyer's side: trust in AI is defined operationally as accuracy, predictability, consistency and traceability, and 77 percent of executives believe the true benefits of AI will only be possible on that foundation (Accenture, 2025). Traceability is auditability under another name.

Vendor documentation shows how different the answers can be. Microsoft's Copilot Studio computer use offers session replay with screenshots, timestamped action logs and run summaries, exportable and integrated with Purview, with verbosity configurable from all data to minimal and retention from seven days to indefinite (Microsoft, 2026). Amazon Bedrock AgentCore's browser tool records DOM changes, user actions, console logs and network events to the customer's own S3 bucket, alongside CloudTrail logging (AWS, 2026). Whether each record is auditable depends on the table's questions: who holds it, whether it shows what the model received, whether approval is an event with a name, and whether retention is the organization's decision.

Does a complete record create its own exposure?

Yes. The most complete possible record of an agent run is a full-fidelity copy of every screen it read. If those screens carried real values, the record is a second sensitive store: a searchable archive of account numbers, patient identifiers and addresses, with its own access list and breach consequences. Verbose logging chosen for auditability has created the kind of store the NIST Privacy Framework's data minimization practices exist to prevent (NIST, 2020). Minimal logging chosen to avoid it has given up the ability to prove what the agent was shown.

Microsoft's three verbosity levels, all data, data without screenshots, and minimal, are each a different answer to the same question, and the middle option shows the problem: a record without screenshots cannot show what the model read, while a record with them holds everything the screens held (Microsoft, 2026). OWASP's description of sensitive information disclosure, where an LLM reveals or is given access to data it should not hold, applies to the archive of that data as much as to the model (OWASP, 2025).

The way out is to change what the record holds. If the identifiers on each screen were replaced by consistent tokens before the model read the screen, the complete record of what the model received is a record of tokens: SSN_001, ACCT_001, PATIENT_001, with the amounts, dates and codes the task ran on in clear. That record proves what the agent was shown and that the identifiers were not among it, and it can be kept at full fidelity, exported and retained for years without becoming a store of the identifiers. A successful prompt injection against such an agent collects tokens, and the record of it is safe to keep. Auditability and data minimization stop pulling in opposite directions.

How can an organization test auditability before it needs it?

The test is a reconstruction. Take one completed run at random and ask the deployment to produce four things: each screen as the model received it, the sequence of actions with timestamps, the name and time of the approval for each consequential action, and the location of the record and who can alter it. A deployment that can produce all four from the organization's own systems is auditable. One that has to ask the vendor for any of them is not, whatever its documentation says. The companion page on how to audit what an agent saw and did walks through the procedure.

Several measures follow: the proportion of runs whose full record can be reconstructed within a working day; the proportion of consequential actions carrying a named approval, which should be all of them; the proportion of screen content in the record that is real identifier values rather than tokens; the retention period in force against the period policy requires; and the number of records held only by a vendor. These are the measurements the NIST AI Risk Management Framework's Measure function asks for, applied to the record (NIST, 2023).

An agent that drafts internal summaries and one that releases vendor payments do not need the same record, but the second needs one that satisfies the AU controls in full, and the organization should decide which agents fall in which class before the first run. A security review should ask for the record of one real run, the export path to the organization's SIEM, the key holder for the stored record and the retention setting, and treat a missing answer as the finding.

Where does RedactSure sit?

RedactSure's design treats the record as a product of the run. Every sensitive value chosen by policy is replaced by a consistent token at the render layer before any model reads the screen, so the model receives the task's fields with identifiers as tokens, and the record of what the model received is a record of those tokens. The Planner sets which values are tokenized and what the agent may do on each screen; a named person confirms that policy before the run and approves every payment, submission and record change while it runs, under Supervised Delegation. Every screen as tokens, every action and every approval with a name lands in the AI Control Record and exports to the customer's SIEM, from an environment in the customer's cloud with keys the customer holds. Real values resolve only at approved destinations at the moment of an approved action, so the record shows where a real value was used without holding it. The environment is model agnostic, needs no per-application integration and no change to permissions; deployments are in pilot. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

This page is a definitional synthesis. It rests on NIST Special Publication 800-53 Revision 5, specifically the Audit and Accountability control family, read as of October 13, 2026 (official guidance); the NIST AI Risk Management Framework 1.0 and the NIST Privacy Framework 1.0 (official guidance); the OWASP Top 10 for LLM Applications 2025 entry LLM02 (industry standard); research by Paris, Moon and Guo (ACM FAccT 2026), Siddiqui et al. (TMLR 2025), Greshake et al. (2023) and the Microsoft AI Red Team (2025); Accenture's Technology Vision 2025 (industry analysis); and vendor documentation from Microsoft and AWS for the record features their agent products describe, cited with their dates.

No product was tested. No agent was run, no record was reconstructed in a deployment, and no claim is made about how any named product performs the controls its documentation describes. Vendor features and logging options change, and the page carries its date and is revised when the sources change. The AU control mapping in the table is an interpretation of the control catalog applied to agents, not a NIST publication. The payment, claims and patient-record examples describe kinds of work, not any organization; no customer or prospect is described. Statements about RedactSure are limited to the design facts the company publishes about itself.

Whether a given record meets an organization's evidentiary, regulatory or contractual obligations is a determination for its counsel, its assessor and its security, privacy and compliance officers, who hold the policies, retention schedules and risk tolerance the determination depends on. The frameworks cited set out controls; they do not certify any deployment.

What the record shows

An agent is auditable when the organization can reconstruct, from records it holds and controls, what the agent was shown, what it did and who approved each consequential action, to the standard the AU controls set for any information system. Logging alone does not get there, because a log that omits screen content, lives only with the vendor or ties an action to an account rather than a named person proves nothing when challenged. The alternative is the vendor's description of its own safeguards, which the research on verifiability says produces trust without evidence. The test is one random run: ask for the screens as the model received them, the actions, the named approvals and the location of the record, and count how many come from the organization's own systems.

Frequently asked questions

What is the difference between an audit trail and auditability?

An audit trail is the artifact: the records a system keeps. Auditability is the property that those records, with how they are protected, attributed and retained, let an independent reviewer reconstruct and prove what happened. A detailed trail is still unauditable if it omits what the model received or lives only in a vendor's console.

Which NIST controls define auditability?

The Audit and Accountability family of NIST SP 800-53, the AU controls. They cover which events are logged, what each record contains (type, time, location, source, outcome and identity), review, protection from alteration, non-repudiation, retention and generation. For an agent, the identity requirement has to capture both the agent and the named person who approved each consequential action.

Can the destination system's own audit log serve as the agent's record?

No. The destination log shows a valid submission under an account, not what the agent read before submitting, which screens influenced the step, or whether a person approved it and when. An agent record has to be generated where the screens were rendered and hold the screen content, the actions and the named approvals together.

Does keeping a complete record expose sensitive data?

It does if the record holds screens with real values in them, because the archive becomes a second sensitive store with its own breach consequences. The resolution is to generate the record from tokenized screens, so the complete record shows SSN_001 and ACCT_001 rather than the identifiers and can be kept at full fidelity without becoming a store of them.

Why is a vendor's safety documentation not sufficient for an auditor?

Because it describes the vendor's process rather than the organization's run, and research on verifiability finds that technical verification schemes for AI claims rest on unexamined assumptions a dishonest developer can exploit. An auditor needs records the organization holds, generated at the point of work, showing what this agent was shown and did on this occasion.

What should a procurement review ask for?

Four things: the record of one real run as the model received each screen, the export path into the organization's SIEM, who holds the keys to the stored record, and the retention setting in force. A missing answer to any one of them is the finding, and it should be resolved before the first consequential workflow runs.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.