Building PII De-identification System for unstructured data in Snowflake Removing sensitive data isn’t always the answer — how to build referential tokenization using Snowflake Cortex I’ve …

Snowflake Community is a platform for users of the Snowflake cloud data platform to share knowledge, ask questions, and collaborate. Readers can learn about cloud data warehousing, data analytics, and data engineering best practices. With forums, user groups, and community events, Snowflake Community provides resources for Snowflake users to connect and learn from each other.

Snowflake Community

A five-stage pipeline for PII de-identification in Snowflake that uses deterministic tokenization instead of simple redaction. Unlike Snowflake's built-in AI_REDACT which replaces PII with category placeholders, this approach uses SHA256 hashing to generate consistent tokens — so the same entity across multiple records maps to the same token, preserving referential integrity for deduplication, cross-dataset joins, and frequency analysis. The pipeline uses Snowflake Cortex (Claude Sonnet 4.5) for LLM-powered entity extraction, followed by value normalization, tokenization, and orchestration via SQL UDFs. Batch processing support reduces LLM costs 5–10x for large-scale workloads. Full implementation is available on GitHub.

Building PII De-identification System for unstructured data in Snowflake

Get Anant Damle’s stories in your inbox