Skip to content

AWS

Read original ↗

Automate custom PII detection at scale with Amazon Macie and Step Functions

Summary

A prescriptive reference architecture from the AWS Architecture Blog demonstrating an event-driven pipeline that automatically detects PII the moment a file arrives in S3. The solution extends Amazon Macie's built-in managed data identifiers with custom data identifiers (regex-based, organization-specific patterns like policy IDs and member IDs), orchestrated end- to-end by AWS Step Functions with zero manual intervention. The architectural centerpiece is a three-bucket data-isolation pattern (raw → stage → scanned) that enforces data lineage by physically separating objects by processing state.

Key takeaways

  1. Event-driven trigger, not polling: EventBridge detects S3 PutObject events and triggers the Step Functions workflow automatically — no scheduled scans, no polling. This gives near-real-time classification at point of ingestion. (Sections: Architecture, Solution overview.)

  2. Three-bucket data-isolation pattern: Raw (unscanned ingest), Staged (active scan in progress, auto-expires after 7 days), and Scanned (validated + reports). Unscanned data never mixes with validated data. Versioning on the raw bucket; auto-expiry lifecycle on stage. This is a physical data-lineage enforcement mechanism — a pipeline-state machine reified as storage topology. (Sections: Architecture, Step 5.)

  3. Per-object classification job: A dedicated Macie classification job is created for each individual object rather than batching. This enables real-time detection at the cost of higher API call volume. The post acknowledges this trade-off: for high-volume workloads, batch multiple objects per job to reduce overhead and stay within the CreateClassificationJob throttle of 0.1 requests/second (1 job every 10 seconds). (Sections: Architecture, Hardening.)

  4. Custom data identifiers: Extend Macie beyond standard PII (SSN, credit card) to organization-specific patterns via regex — e.g., POL-[0-9]{6,10} for policy numbers, MEM-[A-Z]{2}[0-9]{6} for member IDs. Up to 10,000 per account, but a maximum of 30 per classification job. If you need more, split across multiple jobs. (Sections: Step 3, Hardening.)

  5. Step Functions orchestration with built-in retry and wait states: Five sequential states — TriggerScan → WaitForMacie (60s pause) → CheckStatus (poll DescribeClassificationJob) → GetFindings (generate CSV+JSON reports + SNS publish) → MoveFiles (raw → scanned + cleanup staging). Wait-state polling handles asynchronous Macie job completion. Typical scan duration: 15–20 minutes for the demo workload. (Section: Walkthrough, Monitor the pipeline.)

  6. High-severity real-time notification: SNS publishes findings for immediate security-team response when severity exceeds a threshold. Timestamped CSV and JSON reports land in the scanned bucket for audit- ready compliance documentation. (Section: Architecture.)

  7. EventBridge over S3 native event notifications: Chosen for advanced filtering, support for multiple targets, and cross-account routing — all absent from native S3 notifications. (Section: Architecture.)

  8. Production hardening guidance: Multi-tenant isolation via separate stacks per tenant (distinct BucketNamePrefix); SSE-S3 or SSE-KMS encryption; cross-account EventBridge rules for multi-account data sources; CloudTrail data-event logging for full audit trail; SNS data encryption with SSE-KMS. (Section: Hardening measures.)

Architectural diagram

S3 PutObject → EventBridge → Step Functions state machine:
  1. TriggerScan (Lambda): copy to stage bucket, create Macie job
  2. WaitForMacie: 60s pause
  3. CheckStatus (Lambda): poll Macie DescribeClassificationJob
  4. GetFindings (Lambda): retrieve findings, generate reports, SNS publish
  5. MoveFiles (Lambda): raw → scanned, clean stage

Three-bucket topology: - Raw bucket: ingestion landing zone, versioning enabled, objects removed after processing - Stage bucket: active scan working area, auto-expires after 7 days - Scanned bucket: validated objects + timestamped compliance reports (CSV + JSON)

Operational numbers

Metric Value
Macie custom identifiers per account Up to 10,000
Custom identifiers per classification job Max 30
CreateClassificationJob throttle 0.1 req/s (1 job / 10 s)
Demo scan duration 15–20 minutes
Stage bucket expiry 7 days

Caveats

  • No production scale numbers: The post is a reference architecture + CloudFormation walkthrough, not a production retrospective. No throughput, cost, or latency benchmarks disclosed.
  • Per-object job overhead: Creating a Macie job per object is explicit design — it gives real-time detection but doesn't scale cost-efficiently for high-volume ingest (acknowledged; batching recommended for production).
  • Macie scan latency is variable: "Scan duration varies from job to job. The amount and type of data being scanned, its compression, and the number of data identifiers used all dictate how long a job takes to run."
  • Non-prod sample code: Explicitly stated as non-prod without the listed hardening measures.

Source

Last updated · 590 distilled / 1,788 read