Data Report · RedactSure Research
How Do You Audit What an AI Agent Saw and Did? The Five Artifacts, and the Log That Holds No Secrets
With five artifacts the architecture produces as it runs: the setup record, the exposure policy, the run history, the approval trail and the change log, all recorded as tokens and exportable to the SIEM. Auditing an agent differs from auditing software or people because the auditor needs both halves of a new question, what did it see and what did it do, answered at machine scale without the audit trail itself becoming a copy of the sensitive records. An agent working under render-layer tokenization and Supervised Delegation produces exactly that trail: complete on every run, empty of every secret. This page lays out the artifact set, the trap most AI logging falls into, and the walk an examiner actually performs. The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company.
Key findings
- Agent auditing has a problem classical logging never had: a complete log of what a model read is, in most architectures, a second repository of the organization’s sensitive records, sitting in a monitoring stack nobody scoped for custody.
- The token-log property resolves it: when the model reads tokens, the complete record of every screen and prompt contains no sensitive value, so the SIEM ingests everything and holds nothing.
- The frameworks converge on requiring what the artifacts provide: NIST’s AI RMF Govern and Measure functions, OMB M-25-21’s risk-management practices, the NAIC bulletin’s written program and accountability structure.
- None of the five artifacts is a report written about the controls; each is the control’s own exhaust, which is what separates evidence from attestation.
- The auditor’s two opening questions are the board’s two questions in professional dress: show me what this agent could see, and show me who approved what it did.
Why is agent auditing a new problem?
Auditing people produced one kind of trail: sparse, consequential, and attached to names by login. Auditing software produced another: dense but mechanical, systems executing code that engineers could read. An agent is neither. It works at software speed with something like human breadth, reading whole screens, making judgment-shaped choices, drafting and acting across systems, and the audit questions that follow are both of the classical ones at once, plus one neither tradition had to ask: what did it see?
The sight question is new because the answer used to be trivial: a person saw what their permissions rendered, and nobody logged the screen. For an agent the question is load-bearing. Its context is its attack surface, under the injection risk OWASP ranks first; its context is also the exposure an examiner under the minimum necessary standard, FERPA or PCI scope needs bounded. An audit that cannot reconstruct what entered model context cannot answer the first question a records regulator asks after any incident.
Which runs directly into the trap. The obvious fix, log everything the model saw, recreates the problem it documents: a plaintext log of every screen an agent read across the claims queue or the patient accounts is itself a records system, richer than most of the databases the controls protect, retained in a monitoring stack with its own vendors, backups and access lists. Security teams that build it have quietly doubled their custody problem in the name of visibility. The way out is not less logging. It is logging that holds tokens, which requires the tokens to exist before the log does, at the render.
The five artifacts
| Artifact | What it contains | The audit question it answers | Produced by |
|---|---|---|---|
| Setup record | Applications connected, credentials used, supervisor named, dates | Who created this delegation, and what can it reach? | The deliberate grant under Supervised Delegation |
| Exposure policy | Field-by-field decision: what stays clear, what tokenizes, signed by the workflow owner | What was this agent allowed to see, and who decided? | The what-should-the-AI-see session |
| Run history | Every screen and prompt as the model received them, tokens included, in order | What did it actually see and do, run by run? | The environment’s recording at the render |
| Approval trail | Every consequential action with approver, timestamp and the file as the approver saw it | Who authorized each payment, submission, record change? | The policy-triggered gates |
| Change log | Policy revisions, supervisor transfers, delegation endings | How did the governance itself evolve? | The same recorded machinery, applied to its own changes |
Two properties of the set carry the audit value. Completeness with emptiness: the run history is total, every read reconstructable, and contains no sensitive value, because USER_001 and ACCT_001 are what the model received. The examiner replays the run without the replay being a disclosure, and the SIEM export inherits the same property. And exhaust rather than authorship: nobody wrote these artifacts for the audit; the mechanisms emit them by operating, which is why they cannot be backfilled, prettied or forgotten, and why their absence in a rival architecture is itself a finding. Assembled per workflow, the five artifacts are what RedactSure calls the AI Control Record: the runtime system of record for how sensitive AI work occurred.
What do the frameworks and examiners actually ask?
The artifact set was designed against operating requirements, and the mapping to the oversight regimes is direct enough to table in prose.
NIST’s AI RMF asks organizations to govern with defined accountability and to measure AI system behavior; the setup record and approval trail are the Govern evidence, the run history is Measure made literal. OMB M-25-21 requires risk-management practices for high-impact federal AI; the five artifacts are those practices’ paper trail, walked in agency form in the Privacy Act article. The NAIC bulletin’s written program and governance accountability structure, adopted across half the states, presume exactly the setup, policy and approval records; the examiner’s file walk with a claim number in hand is staged in the payment-approval article. Internal audit’s five standard questions, and the answers the record returns for each, are worked in the same article’s audit section.
The examiner’s actual walk, whichever regime sends them, has a reliable shape. Pick a consequential outcome, a payment, a submission, a determination. Pull its approval: name, time, the file as approved. Open the run behind it: every screen the agent read, as tokens, in order. Check the run against the exposure policy: did anything enter context the policy excluded? Check the policy against its signature and review date. Five steps, five artifacts, no interviews required to establish the facts, which converts the audit conversation into what it should be: a review of the named person’s judgment, not an archaeology of what the system might have done.
What does the SIEM integration change?
Exporting the trail to the organization’s own monitoring closes the loop that keeps agent governance from becoming a vendor silo.
Operationally, the security team watches agent activity in the console where it watches everything else: runs, gates, declines and takeovers land as events, correlated with the rest of the estate. The prompt-injection walk-through ends exactly there, with the attack attempt read out of the SIEM the next morning as tokens, an incident review without a breach disclosure.
Structurally, the export means the audit evidence lives under the organization’s retention, access and custody rules rather than the vendor’s, and the token property is what makes that safe: the monitoring stack ingests the complete history of every agent’s sight and action while adding nothing to the organization’s stock of exposed records. The vendor, holding ciphertext it cannot decrypt with keys the customer keeps, is not a second place the auditor must visit to establish what happened. One estate, one trail, no secrets in it.
What the record shows
Auditing what an AI agent saw and did takes five artifacts the architecture emits as it runs: the setup record naming who created the delegation, the exposure policy naming what it may see, the run history holding every screen as tokens, the approval trail naming who authorized each consequence, and the change log tracking the governance itself. The set answers the sight question no classical audit had to ask, without the trail becoming a second repository of the records it documents, because tokenization at the render reaches the logs the same way it reaches the model. The frameworks now converging on agent oversight, NIST’s, OMB’s, the NAIC’s, ask for accountability and measurement this trail supplies as exhaust rather than attestation. The examiner’s walk runs five steps from outcome to policy without an interview, and the two questions every board now asks are answered from the same records: the exposure policies say what the AI could read, and the approval trail says who answered for what it did. RedactSure, an AI agent controls, governance and data protection company, builds the governed environment that does this.
Frequently asked questions
Can we audit agents we run on other platforms this way?
Only to the extent those platforms record sight and consequence, and most record neither as tokens: their logs are either incomplete or plaintext. The five-artifact set is a property of the architecture, which is why the tether test puts what exists today as its fifth question.
Who should have access to the run histories?
Broader access than plaintext logs could ever safely get, which is the point: the histories hold tokens, so supervisors, security and auditors read complete runs without a records exposure. Resolution rights stay where they always were, in the systems of record under existing permissions.
How long should the artifacts be retained?
Under the organization’s existing schedules for the workflow’s records class; for agencies, the federal records analysis in the Privacy Act article applies. The token property removes the pressure to shorten retention for custody reasons.
What does a sampling approach look like for high-volume agents?
Same as any high-volume control: sample runs against policy per period, review all gate declines and takeovers, and full-trace any incident. The completeness of the underlying record is what makes sampling defensible.
Does the model’s reasoning get audited too?
The record captures what the model received, produced and did, which is what oversight regimes ask. Judgments about output quality belong to the supervisor’s review at the gate, and the approval trail shows that review happening.
What is the first artifact to ask any agent vendor for?
A run history from your own workflow, with the question this series asks everywhere: show me what the model received. Whether the answer is complete, and whether it is tokens, sorts the audit story in one document.
Related reading
What Is Supervised Delegation? · Who Approves When an AI Agent Is About to Pay a Claim? · What Are the Two Gaps AI Agents Opened in the Security Stack? · What Is an AI Control Record? · On the RedactSure blog: Accountable AI and Workflow Governance
Sources
Frameworks and regulation
- NIST, AI Risk Management Framework. https://www.nist.gov/itl/ai-risk-management-framework
- OMB, Memorandum M-25-21, “Accelerating Federal Use of AI through Innovation, Governance, and Public Trust” (April 2025). https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-21-Accelerating-Federal-Use-of-AI-through-Innovation-Governance-and-Public-Trust.pdf
- NAIC, adoption map for the Model Bulletin: Use of Artificial Intelligence Systems by Insurers. https://content.naic.org/sites/default/files/legal-adoption-map-ai-model-bulletin.pdf
Standards
- OWASP, LLM01:2025 Prompt Injection, Top 10 for LLM Applications 2025. https://genai.owasp.org/llmrisk/llm01-prompt-injection/
RedactSure documents
- RedactSure, “Accountable AI and Workflow Governance” (2026). https://redactsure.com/blog/accountable-ai-and-workflow-governance/
- RedactSure, “The Two Gaps AI Agents Opened in Your Security Stack” (2026). https://redactsure.com/blog/two-gaps-ai-agents-opened-in-your-security-stack/
- Product behavior described on this page reflects RedactSure’s current design.
Bring your hardest questions.
A 25-minute AI Agent Security Review with the founders: threat model, token design, egress paths, audit schema. Or a 25-minute demo on a workflow like yours, with the data hidden from the AI and a named person approving what matters. We come with diagrams, not a pitch deck.
Book a security review Book a demo · Something elseAbout the author
Chris Sowa is a founder of RedactSure and a former CEO of AI companies; he started his first years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.