What is Data Tokenization?
Data tokenization is a method of data protection that involves replacing sensitive data with a unique identifier or "token". This token acts as a reference to the original data without carrying any sensitive information. These tokens are randomly generated and have no mathematical relationship with the original data, making it impossible to reverse-engineer or break the original values from the tokenized data. The original sensitive data remains confidential in a different isolated location referred to as a "token vault" or "data vault".
Thus, tokens are a kind of ‘links’ or ‘pointers’ to the actual data, but by themselves they’re worthless.
How does Data Tokenization Work?
Consider a credit card number (PAN or primary account number) – a highly sensitive piece of information. In the tokenization process, the credit card number is substituted with a randomly generated token. This token value itself bears no connection to the original credit card number and can only be retrieved with the help of a look up table (aka the tokens table). Let’s review the workings of this process in detail:
- Optional Data Classification: This involves identifying the sensitive data that requires protection in a customer data record. This could include credit card numbers, PINs, PII or any other confidential or sensitive information.
- Token Generation: Once the sensitive data is identified, the tokenization engine generates a random token for each item of sensitive data. These tokens tend to be made of alphanumeric plus digit strings with no apparent pattern or meaning.
- Token Mapping: The produced tokens are then linked to the original sensitive data in a key-value map. This mapping information is safely preserved in a database and is referred to as a "tokens table".
- Data Storage and Security: The sensitive data is then relocated to a safe place, an isolated and segregated vault which holds the tokens table. This data vault is typically extensively protected with strong security features such as encryption and granular access controls, auditing, and more. Therefore, the original customer data record is now clean of sensitive data.
- Tokenized Data Usage: When the application requires access to the sensitive data, it sends the token rather than the real data (because it doesn’t have it to begin with). The tokenization engine inside the vault then searches the token table in order to get the original sensitive data, given the system has the permissions.
If you're looking for a data tokenization solution, you can use our Vault which provides it with simple APIs, signing up is free.
What is De-Tokenization?
In the context of data tokenization, it is also important to understand what de-tokenization means. It is the operation of retrieving the original data referenced by a token at runtime. This process is essential when authorized users require access to the actual sensitive information for objectives such as runtime data process, transaction processing, data analysis, or report generation.
To ensure that only authorized individuals may obtain the original data, de-tokenization requires privileged access to the token vault. The vault then looks up the token in the tokens table and returns the original data. It’s fairly simple.
Benefits of Data Tokenization
Data tokenization offers numerous great benefits for data security and protection. Here are the key benefits of data tokenization:
1. Reduced Impact of Data Breaches
The biggest advantage of using tokens for storing sensitive data, like PII or credit cards, is that the impact of a data breach is now drastically minimized. This is thanks to the fact that the sensitive information is proactively stored, separated from the application’s databases.
And how does that work? In some situations working with tokens as a replacement for unique personal identifiers between systems is practical, effectively eliminating the need for accessing the real data. For instance, a backend system can query over two SQL tables using the same (deterministic) token imitating a customer’s email address, without breaking functionality or leaking any information by the tokens.
2. Tokenized Data is Harder to Steal
Even if attackers obtain access to the application’s databases, they will only have the tokenized version of the data, and thus they will be unable to read or use it without the accompanying vault’s access. So with that, you have a great new bar for stealing your sensitive data, because attackers will have to also obtain access to the tokenization system. It is one more system to hack.
3. Cross System Data Security
Tokenized data can be used across multiple systems and platforms. This advantage enables businesses to safely communicate tokenized information while preserving elsewhere the original data from unauthorized access.
4. Security Access Policies
The tokenization engine can be configured to allow access to the original data for specific entities, and it therefore allows you to have more control over the data in your system or between systems. Every time a system accesses the original data, they will have to go through an authorization phase, where data access security policies will be taken into account for either approving or denying the request.
It also lets you update these policies at any time separately from the application’s code, thus bringing more flexibility to managing data security.
5. Lifetime Control
Tokens can have a restricted duration, thereby improving data security. Tokens can be configured to expire after a specific period, decreasing the window of opportunity for unauthorized access to sensitive data.
6. Data Management
Tokenization helps you manage your data in a centralized way. You need to configure once how sensitive data is accessed (through its token) even if it’s stored in multiple systems. Because every time it will be detokenized the tokenization engine will act upon relevant policies to decide how to serve the sensitive data.
Unlike encrypted blobs, tokens are more like objects, thus making them easier to manage in real life. You can search over them on the original values, since they are stored in one place. Tokens can also be scoped, as in a namespace, so you have more control over working with them under a few systems altogether. In addition, you can delete a token from the tokenization engine (single operation), instead of visiting and deleting the token in all the databases (multiple operations).
The bottom line is that tokens are pretty easy to manipulate and thus very effective at solving different data tokenization problems.
Data Privacy Regulations and Tokenization
Businesses are facing greater obligation to safeguard sensitive information as data privacy standards continue to change and become stricter. Data privacy regulations and tokenization are closely connected, as tokenization can be a valuable tool for businesses to comply with these laws while enhancing data security.
One such regulation that emphasizes data protection is the General Data Protection Regulation (GDPR) in the European Union. Data tokenization assists businesses in meeting GDPR compliance by limiting the quantity of personal data identifiers processed, lowering the risk of unauthorized access or exposure.
What is Pseudonymization and Why is it Recommended by GDPR?
Pseudonymization is a data protection method that replaces identifying information with pseudonyms, which are essentially fake identifiers. This makes it more difficult to link the data back to the individual it belongs to.
It adds an extra layer of data security, especially when data must be stored at-rest or shared with other parties while complying to data protection laws.
Stolen de-identified data that an attacker cannot link back to the data subject, doesn’t require a breach notification to your customers!
Pseudonymization of Customer Data
Pseudonymization is the practice of tokenizing PII of customer data. It involves removing all PII data by substituting it with tokens, eventually resulting in de-identifying the customer record. If stolen, it lacks all vitality to it in terms of privacy. This technique is recommended by GDPR.
Right to be Forgotten Implementation Simplified
If you tokenize all PII in a customer record (pseudonymization), and you delete all the relevant tokens of that customer, the record can never be re-identified and can be left as an anonymized record. You can delete a person from the system by deleting all tokens and their corresponding original data from the tokenization engine itself. Leaving stale tokens in other databases is okay, as it won’t reference any existing data.
Token Types and Their Security Strength
There are a few common token types that all tokenization engines support. However, their implementation may vary in how they generate the actual token value itself. And the devil is in the details here. The strength can be between super secure to somewhat secure, but in all cases they are better than holding the original data, of course.
Let’s examine the popular token types, their common use, and how they are logically generated. And most important is to understand how secure each type is, given an attacker, with somewhat unlimited resources, tries to deduce the original values from holding tokenized data (a la the tokens) only.
Description of the Various Token Types
Generation Logic Behind Each Token Type
Security Strength Explained
Advantages of Tokenization Over Encryption
This is honestly a very long debated topic as there are pros and cons to each method. It really depends on the use case.
But first things first, let’s agree on the purpose of each method:
People sometimes use them interchangeably, but it’s not always possible. You cannot tokenize data as a replacement for data in-transit encryption (TLS), right? But on the other hand, you can encrypt sensitive data before storing it inside a database.
Eventually, it all comes down to data access controls, and, generally, both methods greatly serve that purpose! So how come tokenization is better in reducing scopes and controlling access?
In this section we’re going to explore practical advantages:
1. Post Compromise Data Protection
In case of data breaches, since tokens hold no inherent value or meaning, tokenized data remains protected as it yields no usable information. Whereas, if attackers manage to obtain the encryption key, they can unlock the encrypted data and access its contents.
A classic real world example here is the 2019 Capital One data breach that exhibited how tokenization helped in reducing the impact of the cyberattack and securing critical customer data. This incident highlighted the effectiveness of tokenization as a post-compromise data security solution as the attackers cannot just reverse-engineer the tokens and access the original customer data.
2. Reduced Data Exposure
Encrypted data retains some meaning, making it possibly susceptible to decryption attempts. Tokens, on the other hand, are non-exploitable arbitrary strings that ensure that even if a breach occurs, the information stolen remains unusable.
3. Single Source of Truth
In practice, you should strive to have exactly one copy of the PII data in the backend system, and that’s inside the tokenization engine. Single source of truth is a good practice both from a security standpoint and saving storage space.
4. Granularity of Control
The following fact proves it - normally there’s one key to rule them all - used to encrypt high volumes of data. The encrypted data can only be decrypted, c’est tout.
While with tokenization, each token is an object, with its own permissions, tags, expiration and other fields attached to it. It’s built naturally that way, because there’s a mapping table.
5. Reduced Scope of Sensitive Data
One big compliance-related legal issue with encryption is that it doesn’t reduce scope of sensitive data. Imagine you encrypt all PII data in a customer record and you store it in another system. That system is considered part of the scope of compliance that you have to monitor and work on within your security and privacy process. Using tokenization, this system would not become part of the scope.
Apparently, lawyers agree that with enough computing power an attacker will be able to break it and retrieve the original sensitive PII data. Or, also likely by finding bugs in the implementation of encryption.
And honestly, we all know it’s true seeing the last decade of cryptography bugs in SSL, TLS and WPA protocols, to name a few examples.
6. Simplicity and Peace of Mind
Implementing a production-grade tokenization engine is still hard and yet way easier than a full-fledged encryption engine. Encryption algorithms and implementations have seen a big share of attacks over the years and you never know who and when someone might find an issue in your system. Using encryption is one of the most error prone things out there.
7. Token Flexibility
Perhaps one of the biggest advantages of tokenization over encryption is that tokens are designed to answer specific formatting needs, making them versatile for being used in legacy applications that validate for some specific input format (e.g. a token that looks like an email address). Whereas encrypted data typically maintains a fixed and mostly uncontrolled format.
Tokens have the following great characteristics:
- A token can conform to any predetermined data format, like an email address standard.
- A token can be of arbitrary length, as long as there are no collisions produced.
- A token can be created to easily preserve sortability, although it might leak data in certain situations.
It’s pretty much next to impossible to have such elasticities with encryption.
8. Reduced Storage Space
Remember that by keeping the token value length short you can really save a lot of space when you have a lot of PII data, both in RAM, disk and CPU processing it inside a database. In high data volumes this can seriously become an advantage in an economical world.
Compared to encryption, it’s impossible to reduce the encrypted blob size. It often gets padded and also contains more helper fields.
9. Scalability
Tokenization methods are often easier and faster than encryption, which include complex mathematical algorithms. Tokenization entails token mapping and extraction, but encryption demands both encryption and decryption steps, which can be expensive for applications on a large scale.
A word of advice - in reality, this assumption might be wrong. It’s really up to how you implement your system and how you use these methods. Without benchmarking performance, it’s hard to conclude.
10. Deletion of Data
One of the most important advantages of using tokenization is that it provides you with extra granularity of control. When you want to delete references (tokens) to the original data in certain situations, it’s very simple. Imagine there’s a customer record and you tokenized its PII data, and now you want to get rid of this PII data alone. You just delete the relevant tokens, without even accessing the customer record that is stored elsewhere, or over multiple data stores, in real life.
You cannot do this with basic encryption - in encryption you would need a whole crypto-shredding mechanism implemented. It takes a huge toll to enroll a key per person (or encrypted PII element) and it becomes very challenging to implement and maintain, while tokenization is ten times simpler.
11. Irreversibility
Tokenization, unlike encryption, is a one-way function. Tokens have no fundamental connection to actual data, making it impossible to reverse-engineer the sensitive information from the tokenized data alone.
In payment systems, sometimes there’s a need for a PCI one-way token - that is taking a credit card number and having a one-way hash to it, without collisions or other issues. Tokens can do that in a purely safe manner, by also locking the ability to access the original data.
Integration of Data Tokenization vs Encryption in Your Code
After we have covered all the reasons to use tokenization, we have to play fair, and therefore, it’s worth mentioning that there are two big advantages to encryption in general:
- Encryption protects the data in-place
- Encryption can run anywhere (which sometimes is a minus, because of potential key compromise)
- Encryption is stateless
The bottom line is that encryption truly has one big plus when it comes to code integration, and that’s because it’s stateless.
Step 1: You can encrypt the data.
Step 2: Stuff it to a database.
Step 3: If there was some error with the insert query, you don’t need to clean up encrypted data in any way.
In tokenization, you have to work against two stateful components, and transactions have to be done in a safe manner and cleaning up should clean orphaned tokens. Nevertheless, tokenization still wins when you use it across your whole system and not sporadically to protect some data fields.
What Makes Tokenization Especially Strong
It’s all about the mechanism of how tokenization works.
Tokenized data, when accessed or read from a database, is not automatically detokenized!
And this fact plays a key role in reducing the impact of data breaches. Whereas encrypted data is automatically decrypted upon accessing the data, thus transparent and therefore sometimes quite useless against attacks.
Tokenization is really an extra layer of defense because the detokenization is separated and happens only at runtime!
With tokens being a replacement for unique identifiers (like emails, phones, national-id) it’s possible that systems will not even work with the sensitive data at all, thus further reducing scope of sensitive data.
Why are Businesses Increasingly Adopting Tokenization as a Security Measure?
Tokenization's ease of implementation, integration, and scalability along with the other mentioned advantages and features, contribute to its widespread acceptance across a range of industries. Tokenization has, ultimately, become an essential practice in today's data-driven world, offering a powerful way of protecting valuable information and mitigating the dangers posed by data breaches.
The bottom line is that tokenization serves as an effective extra security defense layer!
There are several reasons why businesses have increasingly adopted data tokenization in recent years. These include:
Stringent data regulations: Governments around the world are increasingly imposing strict data regulations, such as the General Data Protection Regulation (GDPR). These regulations require businesses to take steps to protect sensitive data, and recommend pseudonymizing data.
Third-party risk mitigation: Many companies entrust third party vendors and partners and have to provide access to customer data. The use of tokenization can reduce the risk of data exposure, as only tokenized data is received by the vendors.
Customizable security levels: The level of security required for different types of data can be customized through tokenization solutions. This flexibility guarantees that more sensitive information is safeguarded at a higher level.
Fraud and identity theft prevention: Tokenization reduces the risk of useful data being stolen. It is used for mitigating fraud since stolen tokenized data doesn’t have the context needed for successful fraud attempts, eliminating identity theft risk.
When Should Businesses Use Data Tokenization?
Businesses should majorly consider using data tokenization in the following scenarios:
- Cloud-Based Applications: Data tokenization can help businesses have an extra yet most-critical defense layer in addition to the in transit and at rest encryption. It assures that regardless of a security breach, the sensitive data remains inaccessible without the right permissions, which requires yet another step for the attackers.
- Data Analytics and Testing Privacy: Tokenization enables businesses to use authentic but unidentified data for data analytics and testing without revealing genuine customer information, hence protecting data privacy.
Businesses don’t have to compromise on their customers’ privacy when giving analysts access to the data. They can enable innovation by encouraging the use of a PII-free version of it.
- Personally Identifiable Information (PII) Handling: When dealing with personal data like names, addresses, emails, phone numbers, or medical records, data tokenization can help maintain privacy and compliance with data protection regulations. Because once a customer record goes through a data tokenization process (pseudonymization) it becomes de-identified, thus if stolen, the impact on the customer is truly reduced.
- Payment Processing: Companies can protect customers by tokenizing credit card numbers to reduce the PCI scope, lowering the risk of payment fraud and data breaches.
Identifying Sensitive Data for Tokenization and Understanding the Exposure Risks
Identifying which sensitive data should be tokenized is different on a per business case. For example, financial institutes would want to tokenize PII and some financial information. Whereas health-care businesses would prefer to tokenize PHI and some medical records, etc.
Determining Data Sensitivity
Determining data sensitivity involves analyzing the level of risk associated with specific types of data. Not all data can be considered sensitive, and its significance changes depending on the circumstances and activities of the business.
Our guidelines are:
- Normally, the minimal amount of information tokenized that makes the rest of the data de-identified is a good balance.
- If the sensitive data is stolen and can cause harm or violate privacy of the data subject, then it’s probably something we want to protect.
An example about practically what to tokenize or not:
Suppose you have a monthly pay-slip, with the mentioned salary, the breakdown and other sensitive information. It would be sufficient to tokenize the social security number and the full name (or other PII - personal identifiers, if they exist). Now, the newly de-identified pay-slip is stored somewhere without any identifiers, and thus can be considered safe.
The salary by itself is not interesting any more as long as the pay-slip is anonymized, so it’s not necessarily important to tokenize the salary per se. In case you want to do aggregated calculations on the salaries of all employees, and even if they’re anonymized, it doesn’t disturb this process.
This is a bit counterintuitive as people are used to treating salaries as the most important asset, but technically we do it the other way around, as follows:
You can learn more about this process in our PII cheat sheet.
PCI, PII, PHI, and Other Categories
Payment Card Industry (PCI-DSS): PCI data covers credit card numbers, CVV codes, and cardholder names, expiration dates, which should be tokenized in order to guarantee compliance with industry standards. It is classified as highly sensitive information because its disclosure could result in financial losses and potential fraud.
Personally Identifiable Information (PII): PII refers to any data that can identify an individual through information such as name, phone number, home address, email address, social security number, social media handle, etc. PII is also regarded as extremely valuable because its publication, i.e. data theft and then leaking the data, can result in fraud or other violations of privacy.
Protected Health Information (PHI): PHI includes medical records, treatment details, lab tests results, and other additional medical information that requires protection under the Health Insurance Portability and Accountability Act (HIPAA). Unauthorized sharing of PHI can have serious legal repercussions for both individuals and healthcare providers.
Other categories: Intellectual property, financial records, and confidential business information fall into the other categories of sensitive data. Through this article, as you know already, we mostly focus on transactional structured data in applications.
The Risks of Exposing Sensitive Data
The consequences of revealing confidential data can be catastrophic. Financial losses, lawsuits, and reputational damage can all result from data breaches. In addition, businesses that fail to effectively protect sensitive information may suffer non-compliance fines and sanctions.
It's essential for businesses to implement proper data classification, encryption and tokenization, and appropriate access controls to safeguard sensitive data effectively. This will help reduce the impact of the damage and above risks.
Choosing the Right Data Tokenization Solution: Key Metrics
It is critical to use the correct data tokenization solution to provide strong data safety and adherence with data protection standards. Here are key metrics to consider when evaluating data tokenization solutions:
Integrating Data Tokenization into Data Security: Different Use Cases
Tokenization in Databases and Data Storage Systems
Data tokenization at the database level provides a holistic approach to data protection. Businesses can assure data security at rest by tokenizing confidential information directly in the database, protecting against data breaches even if unapproved individuals get access to it.
In the aftermath of a data breach, the stolen tokenized data remains devoid of value or meaning. Tokens are impossible to reverse engineer to the original data, making the stolen data useless to cybercriminals. This post-compromise measurement is a unique feature of tokenization, giving businesses the assurance that even if their safety measures are compromised, their data will be safely secured. By adopting tokenization directly into the database, businesses provide an extra layer of defense, protecting critical data even in the case of a breach.
The advantage of utilizing tokenization as an extra layer of protection on top of databases is that it’s agnostic to the database implementation and basically it’s easy to use it with any data store solution. This makes tokenization a compelling solution that can organizationally scale for holistic data access controls.
Tokenization for Data in Transit
Data must be protected while in transit to avoid interception and unauthorized entry. Data can be tokenized before it leaves its original security boundary, ensuring that classified data is protected during transmission.
Tokenization for data in transit complements other security measures such as encryption (TLS and HTTPS protocols). While encryption protects data at rest and in transit, tokenization is an additional layer of security specifically to protect sensitive data during transmission. Some organizations have a policy that dictates that PII data must be tokenized when it moves between systems/boundaries.
Tokenization in Data Warehouses / Data Lakes
By tokenizing sensitive data before it enters the data warehouse, businesses can protect their customers' privacy while also driving innovation. Assuming all PII data is tokenized, then the rest of the data is now de-identified, and letting data analysts and data scientists access it will enable more innovation for the business. This promises securing customers’ privacy, which is important because data warehouses are usually accessible to many employees who download the data and work with it on their machines.
Tokenization in Cloud Environments
Data security is an ongoing concern for businesses as they migrate their data and applications to the cloud. Tokenization may be easily implemented into cloud-based systems, providing a secure method of protecting sensitive data in an environment that is shared. PCI Tokenization is widely used in payment processing environments, including cloud-based payment processing services.
PCI Tokenization is a specific implementation of tokenization that is designed to comply with the Payment Card Industry Data Security Standard (PCI DSS). It is a collection of security standards designed to protect cardholder data and ensure credit card transaction security.
Data Tokenization Best Practices
Tokenization of data is a powerful technique when implemented correctly. To ensure the effectiveness of the tokenization solution, consider these best practices:
- Random Tokens: Prefer to use completely randomly-generated tokens as much as possible, as they provide the maximum security, unlike deterministic tokens that might leak information about the concealed original data.
- Limit Data Exposure: Tokenize just the data that needs to be protected, lowering the total amount of sensitive data in the system.
- Secure Token Storage: To prevent unauthorized access, provide strong security measures for the token vault, such as encryption, access controls, and monitoring.
- Regular Audits: Conduct regular audits to confirm that the tokenization solution is still viable and meets data security requirements and regulations.
- Employee Training: Educate employees about data tokenization, its benefits, and their role in maintaining data security.
Raising the Security Bar with Tokenization
The bottom line is that if attackers manage to penetrate into an application’s backend network and gain access to databases, they won’t be able to access the sensitive data, given that it was tokenized. Normally, the attacks we see are the ones that don’t manage to run arbitrary code in the backend system, making them useless against tokenized data!
Create a free Vault account right away, tokenize your sensitive data by using our APIs.
Attackers usually only have some network access and therefore can hop to some databases, if they managed to put their hands on the DB’s credentials. With tokenization applied, the attacker will have to run code in the system, accessing the tokenization engine through its APIs and starting to detokenize lots of data. This is a substantial extra step in accessing the sensitive data, and normally attackers don’t have such access to the system.
Now attackers will have to obtain the credentials to the tokenization engine too. Each step for accessing the sensitive data slows them down and is another barrier making their life harder, which is everything we care about.
Supposedly they find a way to run code, it gives an opportunity to the security teams monitoring everything to find this incident while it’s being carried out. If you were wondering why encryption isn’t so useful sometimes, it is because most encryption solutions are designed to be automatic and transparent, thus in practice they don’t stop attackers from accessing sensitive data.
Conclusion
In conclusion, data tokenization is one of the best techniques to defeat data breaches. Its one-way conversion of sensitive data into indecipherable tokens provides exceptional safety, shielding businesses and their customers from cyber threats. The implementation is pretty straightforward and bulletproof. The integration is definitely easy too. As the digital world advances, adopting this cutting-edge security measure is not only a visionary option, but a requirement.
Businesses can strengthen their data defenses, comply with regulations, and gain the trust of their stakeholders by incorporating data tokenization. Its advantages go far beyond its strong security measures. The scalability, efficiency, and its convenient integration into systems make it a preferred choice for all businesses - paving the way for a future where sensitive information remains truly invulnerable.
Gil is a software ninja who loves both building software (companies too) and breaking code. Renowned for his prowess in security research, including notable exploits of the Microsoft Windows kernel that have earned him unusual high bounty awards. He has written a couple of very successful open source libraries. And he likes to talk publicly in conferences.
It all begins with the cloud, where applications are accessible to everyone. Therefore, a user or an attacker makes no difference per se. Technically, encrypting all data at rest and in transit might seem like a comprehensive approach, but these methods are not enough anymore. For cloud hosted applications, data-at-rest encryption does not provide the coverage one might expect.
Senior Product Owner
