Skip to content
redactsure
Book a demo

Explore.

Explainer · By Chris Sowa · Published
Last updated

What is Data Tokenization?

A token stands in for the real value in everyday systems while the original stays in a secured vault, reachable only through the tokenization system. Confuse tokenization with encryption, masking or redaction and a team may believe a value is protected while it still sits in the database, the prompt and the log.

A sensitive browser field becomes a token linked to an isolated vault record.

A token stands in for the real value in everyday systems while the original stays in a secured vault, reachable only through the tokenization system. Confuse tokenization with encryption, masking or redaction and a team may believe a value is protected while it still sits in the database, the prompt and the log. Data tokenization is the replacement of a sensitive value with a substitute value, called a token, that has no exploitable meaning on its own and that can be mapped back to the original only through a controlled tokenization system. The original value is held separately, typically in a secured vault, and ordinary systems work with the token instead. Payment guidance defines tokenization as replacing a card number with a token usable in its place for some purposes, while the real number is stored in a card data vault (PCI Security Standards Council, 2011). The same pattern now applies to account numbers, identity numbers and health identifiers, and in AI work it takes the form of render-layer tokenization, applied before any model reads the screen. The environment that enforces this is built by RedactSure, an AI agent controls, governance and data protection company.

Key findings

  • A token is a surrogate with no mathematical relationship that lets a holder derive the original value from the token alone. Retrieval depends on access to the tokenization system, not on a key held with the data (PCI Security Standards Council, 2011).
  • Encryption differs because ciphertext is the original value transformed by an algorithm and a key. Anyone who obtains both the ciphertext and the key can recover the original, wherever that happens, which is why the PCI guidance treats systems that can reach the original differently from systems that hold only tokens (PCI Security Standards Council, 2011).
  • Masking and redaction differ because they are mainly presentation or publication controls. Masking shows a partial value, and redaction removes or obscures a value in a document or view (NIST, 2023). Neither by itself provides a controlled way back to the original for an approved process.
  • Tokenization systems have defined components: token generation, a mapping store or vault, detokenization under authentication, and monitoring of both processes (PCI Security Standards Council, 2011).
  • Systems that cannot detokenize are in a different risk position from systems that hold tokens only, a scope point the PCI guidance makes for cardholder data environments (PCI Security Standards Council, 2011).
  • NIST de-identification guidance treats replacement with surrogate values as one transformation among several for direct identifiers, and cautions that repeatable replacement can carry re-identification risk if context allows linking (NIST, 2023).

How is tokenization defined in standards sources?

The PCI Security Standards Council tokenization guidelines describe tokenization as replacing a primary account number with a token, storing the number and its token in a card data vault, and returning the token to the requesting application for later use (PCI Security Standards Council, 2011). Three parts of that definition matter beyond payments. First, the token stands in for the value in ordinary processing. Second, the mapping between token and value exists in one controlled place. Third, getting the value back is a separate, authenticated operation called detokenization, not something any holder of the token can do.

A token may be generated randomly or through a cryptographic method, and it may preserve the format of the original so existing systems can store it. The PCI guidance notes that security depends on the generation method, vault protection, and detokenization controls (PCI Security Standards Council, 2011). Tokenization is not truncation, which shows part of the original, or a plain hash of a low entropy value, which can be attacked by trying every possible input. NIST makes a related point for keyed hashing: the key needs sufficient randomness and protection (NIST, 2023).

What are the components of a tokenization system?

A working system has more than a substitution step. The vault and its controls become a high value part of the estate that must be protected and audited.

Component What it does Control question
Token generation Creates or retrieves the token for a value when the value is first presented Is generation unpredictable, and does the same value get a consistent token where matching is required?
Token vault or mapping store Holds the token to value mapping, with the value protected at rest Who and what can access the vault, and is it separated from systems that hold only tokens?
Tokenization interface Accepts a value from an authorized application and returns a token Is the caller authenticated, and is the event logged?
Detokenization interface Returns the original value to an authorized requester for an approved purpose Is each request authenticated, authorized by purpose, and logged, and can it be denied?
Token lifecycle rules Covers expiry, replacement, deletion, and what happens when a value changes Are tokens retired when the underlying value is retired?
Monitoring and audit Records tokenization and detokenization events for review Can unusual detokenization volume be detected and investigated?

The PCI guidelines set out tokenization and detokenization as processes in which the requesting application is authenticated first, the vault is queried, and failures are logged (PCI Security Standards Council, 2011). The token can travel widely because reversal is narrow, gated, and recorded. Vaultless designs change where the mapping lives but not the control questions: wherever reversal depends on a key, that key needs the same discipline as a vault.

How does tokenization differ from encryption, masking, and redaction?

These four controls are often listed together, and they can be combined, but they operate at different layers, at different times, and with different reversibility. Confusing them leads to gaps: a team may believe data is protected because it is masked on screen, while the full value still sits in the database, the prompt, and the log.

Control Layer and timing Reversibility What a holder of the protected form gets
Tokenization Applied when a value is captured or before it is passed to a system or model that does not need the original. The token is used during processing. Reversible only through the tokenization system, under authentication and policy. No useful mathematical route from token to value. A reference that supports matching and workflow steps, but not the value itself.
Encryption Applied in transit, at rest, or at field level, before storage or transmission. The ciphertext is stored or sent. Reversible by design with the correct key, wherever the key and ciphertext meet. Ciphertext, which becomes the value if the key is obtained.
Masking Applied at presentation, when a value is displayed, often showing only a fixed portion. The underlying value usually remains in the source system. Not a reversal process. The full value is retrieved from the source, not recovered from the masked display. A partial view, such as the last digits, enough to recognize but not to use the value.
Redaction Applied to a document, image, screen, or dataset, usually before sharing or publication. The value is removed or obscured in that copy. Intended to be irreversible in the redacted copy. The source, if retained, still holds the value. A copy with the value absent or obscured, and no built in route back from that copy.

Encryption protects confidentiality in storage and transit, and it is essential inside a token vault itself. Its difference from tokenization is capability. An application that can decrypt can reach the original, so it remains in scope for that data. An application that holds only tokens, with no detokenization rights, cannot. The PCI guidance makes scope depend on exactly that capability (PCI Security Standards Council, 2011).

Masking is a display decision. NIST lists masking with a repeating character as one way to remove direct identifiers (NIST, 2023). It does not change what the underlying system, an export, or a model reading the raw record can access. Redaction is a copy decision, appropriate for a document that will be shared in that form, and its weakness is completeness: miss one occurrence and the copy still discloses. Tokenization is a processing decision: it changes what downstream systems hold during work, while preserving an approved route back.

When can a token be reversed, and who should be able to reverse it?

Detokenization should be rare, purposeful, and attributable. The requester should be authenticated, the purpose should match policy, and the event should be recorded. Bulk detokenization, detokenization to a new destination, or detokenization outside normal hours should need stronger approval or be denied, because those patterns match how a token store would be abused.

Separation is the supporting rule, and it matches the zero trust principle that each access request is evaluated on its own rather than trusted by location (NIST, 2020). Systems that use tokens for routine work should not hold detokenization rights as a convenience. Where a workflow needs the real value at an outside destination, the value should be resolved at that destination at the moment of use, not passed back through every intermediate system.

Token consistency also needs a decision. A consistent token allows matching across records, which many workflows need. It also creates a linkable identifier. NIST warns that repeatable transformations produce pseudonyms and can raise re-identification risk (NIST, 2023). Tokenized data therefore needs access control and purpose limits as well, not only the vault.

Where is tokenization used, and what limits does it have?

Payments remain the most documented use. The same structure fits bank account numbers, government identity numbers, health identifiers, and customer account references. In AI workflows, tokenization has a specific use: replacing values before a model reads a screen or record, so the model can work with structure and consistency without holding the values.

The limits are practical. Tokenization adds a system to run and keep available. It does not fix authorization errors: if an approved process is tricked into detokenizing for a wrong destination, the failure is in the approval logic. It does not protect values that enter through another channel, such as a free text note. And it does not make data anonymous. Tokens are pseudonyms, governed as sensitive data in their own right. Nor is tokenization a compliance shortcut: the PCI guidelines state that they do not replace the requirements of the PCI Data Security Standard (PCI Security Standards Council, 2011).

What should be checked before adopting tokenization?

Ask where the value will still appear. Map capture, storage, screens, exports, logs, and model context, and state for each place whether it will hold the value, a token, a masked view, or nothing. Ask who can detokenize, and test that routine users cannot. Ask what happens on exit, including token retirement.

For agent use, add two checks. Confirm that tokenization happens before the model reads content, not in a filter applied to output afterwards. And confirm what a prompt injection would obtain (OWASP, 2025). If the answer is tokens plus task context, with real values resolving only at approved destinations after human approval, the design matches the standards definition of tokenization. If the agent can call detokenization itself, the token layer is thinner than it appears.

Where does RedactSure sit?

RedactSure applies tokenization in its runtime form for agent work. Substitution happens at the render layer, at the moment a screen is prepared, before any model reads it, rather than as storage hygiene applied long before or as output cleaning applied afterwards. The model receives the task's fields with identifiers as consistent tokens, such as ACCT_001 and CARD_001, so it can match records and follow the workflow without holding the values. The Planner chooses which values become consistent tokens per screen, and a named person confirms that policy before the run under Supervised Delegation. Real values resolve only at approved destinations during approved actions, and the agent holds no detokenization rights of its own; every payment, submission and record change waits for a named person.

Customer-held keys mean RedactSure stores ciphertext it cannot decrypt, and each tokenized screen, action and approval enters the AI Control Record, which exports to the customer's SIEM. The environment is model agnostic and needs no per-application integration, and user permissions do not change. Deployments are in pilot. Everyone else controls where data can go. RedactSure controls what the AI can see.

Methodology and limitations

Definitions on this page come from two standards sources: the PCI Security Standards Council's "PCI DSS Tokenization Guidelines" information supplement (2011; official guidance) and NIST Special Publication 800-188, "De-Identifying Government Datasets" (2023; official guidance). NIST SP 800-207, "Zero Trust Architecture" (2020; official guidance), supplies the per-request access principle applied to detokenization, and OWASP's LLM01:2025 Prompt Injection entry (standard) supplies the agent check. All sources were read as of October 13, 2026. The PCI document's process descriptions are used as the reference model for tokenization and detokenization; its age and payment scope are limitations, and newer product designs, including cryptographic and vaultless approaches, are discussed only at the level of the control questions they must still answer.

No tokenization product was tested, no vault was inspected, and no performance or security benchmark is claimed. Vendor features and product designs change, and the page carries its date and is revised when the cited documents change. The examples of account numbers, identity numbers and health identifiers describe data types, not any organization; no customer or prospect is described. Blockchain and payment network uses of the word token are outside the scope here, which covers data tokenization for sensitive values.

Whether a tokenization design reduces scope under PCI DSS, satisfies a de-identification standard, or meets any other requirement belongs to the organization's counsel, its assessor, and its security and compliance officers, not to the page.

What the record shows

Tokenization replaces a sensitive value with a surrogate carrying no exploitable meaning, and reserves the way back for a controlled system mapping token to original under authentication. That design separates it from encryption, which travels with a reversing key, from masking, which shows a partial value, and from redaction, which removes one without a governed route back. Tokens remain pseudonyms rather than anonymity, and detokenization rights decide the real risk position of every system holding them. The test for any system is one question: list every place the value still appears, and name every process that can turn a token back into it.

Frequently asked questions

Is tokenization just another name for encryption?

No, and the difference decides who can recover the original. Encryption transforms a value with an algorithm and a key, so anyone holding both the ciphertext and the key can reverse it wherever they are. A token has no mathematical relationship that lets its holder derive the original; recovery depends entirely on access to the tokenization system and its mapping store. That is why taking tokens out of a system does not hand an attacker a puzzle to solve, only a reference they cannot resolve.

Does tokenizing a value make the data anonymous?

No. NIST de-identification guidance treats replacement with surrogate values as one transformation among several, and warns that repeatable replacement can carry re-identification risk when surrounding context allows records to be linked. A tokenized customer can still be followed across systems because the same token recurs, which is the property that makes tokens useful for workflows. Tokenized data should therefore be governed as sensitive data in its own right, with access controlled and detokenization monitored.

Who should be allowed to detokenize?

Only authenticated, approved processes with a defined need, and the fewer systems that can, the better. The PCI guidance draws a scope distinction between systems that merely hold tokens and systems able to detokenize, because the second group sits closer to the real values and inherits more risk. Routine users and, for agent workflows, the agent itself should not hold detokenization rights; resolution should happen at the approved destination, at the moment of an approved action.

What is vaultless tokenization?

A design in which the token is derived from the value by a cryptographic method under a key, so no mapping table has to be stored. It changes where the reversal capability lives, not whether it exists. Whoever holds the key can reverse any token, so the key needs the same protection, access control and monitoring a vault would need, and the same scope questions apply to every system that can call the reversal function.

Does tokenization take a system out of PCI DSS scope?

Not by itself. The PCI tokenization guidelines state that they do not replace the requirements of the PCI Data Security Standard. What the guidance does say is that systems holding only tokens, with no ability to detokenize, are in a different position from systems that can reach the real number. Whether a particular design reduces an organization's scope is the assessor's determination, made against the full set of systems that store, process or transmit card data.

How does tokenization apply to AI agents?

The value is replaced before the model reads the screen or record, so the model works with ACCT_001 or CARD_001 and task context rather than the real identifiers. Consistent tokens let the agent match records across screens. Real values resolve only at an approved destination at the moment of an approved action, which a named person confirms. If the agent can call detokenization itself, a successful prompt injection can obtain the value, and the token layer protects less than it appears to.

Book an AI Agent Security Review →

Bring your hardest questions.

Bring one sensitive workflow and the decisions your team needs to keep in its hands.

Book a demo

About the author

Christopher Sowa
Chris SowaCo-Founder, President and Chief Commercial Officer, RedactSure

Chris Sowa is a founder of RedactSure and a former CEO of an AI company; he started it years before ChatGPT existed. He previously led AI at Accenture, served as Global VP of Strategy & Innovation at Schneider Electric, was CCO of Sovos, and spent more than a decade at Oracle, with earlier roles at SAP and IBM.

Book a demo

Book a demo

Having trouble? Open the booking page or email us.