Protecting personal data before analytics consumption is a core requirement in modern data platforms. In AWS-based lake architectures, this is typically achieved through data de-identification during ingestion or transformation. This post outlines a practical and production-ready approach to hiding personal information using Spark jobs in AWS Glue.

---

What “Hide Personal Information” Means in Data Engineering

Hiding personal information usually refers to de-identification, an umbrella term that includes:

• Masking: replacing characters with placeholders.
• Redaction: removing sensitive substrings entirely.
• Hashing: irreversible transformation for joinability.
• Tokenization: reversible replacement using secure mappings.

In regulated environments (GDPR, HIPAA), this step must occur before data is exposed to analytics, BI tools, or data science workloads.

---

Why Do This in AWS Glue?

AWS Glue provides a managed Spark environment that is well suited for:

• Large-scale batch processing.
• Schema-aware transformations.
• Integration with AWS-native security services.

By de-identifying data inside Glue jobs, you ensure that only compliant datasets reach curated layers (silver/gold).

---

Reference Architecture

Flow:

1. Raw data lands in Amazon S3 (raw/).
2. A Glue Spark job reads the data.
3. Personal data is detected and hidden.
4. Clean data is written to clean/ or curated/.

Two common implementation patterns are described below.

---

Pattern 1: PII Redaction with Amazon Comprehend (Recommended)

Amazon Comprehend provides managed PII detection and redaction.

How it works

• Use Comprehend’s PII APIs to detect entities such as emails, phone numbers, or names.
• Apply masking or replacement.
• Persist the sanitized output.

When to use

• Text-heavy columns (comments, logs, descriptions).
• When you want managed accuracy without maintaining NLP models.

Official documentation:
https://docs.aws.amazon.com/comprehend/latest/dg/redact-api-pii.html

Pattern 2: Column-Level Hashing in Spark

For structured datasets with known sensitive fields:

from pyspark.sql.functions import sha2, concat_ws, lit, col

SALT = "<secure-salt>"
df_clean = (
    df.withColumn(
        "email_hash",
        sha2(concat_ws(":", col("email"), lit(SALT)), 256)
    )
    .drop("email")
)

When to use

• Deterministic joins are required.
• Re-identification is not allowed.
• Performance must be fully inside Spark.

---

Operational Considerations

• Salts and keys must be stored in AWS Secrets Manager or encrypted with KMS.
• Avoid synchronous external API calls at high Spark parallelism unless rate-limited.
• Prefer pre-ingestion de-identification over post-analytics fixes.
• Log transformations for auditability, but never log raw PII.

---

Conclusion

Hiding personal information in AWS Glue is not a single feature, but a design choice embedded in your data pipeline. Combining Spark transformations with AWS-native services allows you to meet privacy requirements without sacrificing scalability or performance.

De-identification should be treated as infrastructure, not as an afterthought.