Skip to content

PATTERN Cited by 1 source

Three-bucket data-isolation

Definition

Enforce data lineage and processing-state separation by physically partitioning objects across distinct storage buckets according to their pipeline stage. Each bucket represents exactly one state in the processing lifecycle — unscanned data never co-exists with validated data.

The canonical shape (from the 2026-07-22 AWS Architecture Blog PII detection pipeline):

Bucket Role Lifecycle
Raw Ingestion landing zone Versioning enabled; objects removed after successful processing
Stage Active processing / scan in progress Auto-expires (7 days); temporary holding area
Scanned Validated output + compliance reports Permanent retention; audit-ready

Why it matters

  • Compliance: Regulated industries (GDPR, HIPAA, CCPA, PCI DSS) require demonstrable data lineage — knowing where data is in its classification lifecycle at any moment. Physical bucket separation makes this trivially auditable.
  • Blast radius: A misconfigured lifecycle rule, accidental deletion, or access-control error on one bucket doesn't contaminate data in other processing states.
  • Operational clarity: Monitoring and alerting can be scoped per-bucket (e.g., alert if raw bucket object count exceeds a threshold — a sign of pipeline backup).

Trade-offs

  • More S3 buckets = more IAM policy surface: Each bucket needs its own access policies, encryption keys, and lifecycle configuration.
  • Data movement cost: Objects physically move between buckets (copy + delete), not just metadata updates. At high volume this adds S3 PUT/GET cost.
  • Not a substitute for encryption: Bucket separation enforces lineage, not confidentiality — SSE-S3 or SSE-KMS encryption is still required on every bucket.

Seen in

Last updated · 590 distilled / 1,788 read