AWS 2026-10-26

Data governance with annotations in Amazon S3¶

Summary¶

An AWS Architecture Blog post that works through Amazon S3 annotations — a mechanism for attaching governance context (classification, retention policy, jurisdiction/residency, PII inventory, rationale) directly to an S3 object version rather than keeping it in a separate metadata database one system away from the object it describes. A single object version can hold up to 1,000 annotations, each a named UTF-8 payload of 1 byte to 1 MB (JSON/XML/YAML/plain text), so each annotation is a complete governance record rather than a key-value pair. The post lays out an event-driven reference architecture with four paths (write / maintenance / backfill / read) over S3 at the center, then — its real contribution — enumerates the handful of S3 behaviors that decide whether the design is correct: version-scoped deletion, Object Lock sequencing, multipart-copy annotation drop, irreversible annotation overwrite/delete, the SSE-C constraint, separately-scoped IAM actions, and the trust boundary around machine-generated labels. The organizing mental model: an annotation is caller-authored context bound to a specific object version; S3 stores it and moves it, but does not interpret or validate it, and its lifecycle follows the object version's lifecycle.

Key takeaways¶

Governance context travels with the data. Annotations are stored with the object version, not in a separate system, so CopyObject/replication move the classification or residency label along with a DR copy — context doesn't drift from the object it describes. (Source: sources/2026-10-26-aws-data-governance-with-annotations-in-amazon-s3)
Revisable in place without rewriting the object. PutObjectAnnotation updates an annotation without changing the object's ETag — a classification can be corrected, or a generated label promoted from proposed to reviewed, with no object rewrite. The in-place update is itself replicated, so a later re-classification on the source keeps a cross-Region copy current without recopying the object.
Governance answers without reading the data. A read-only-annotation consumer can answer "which objects hold PII and have an expired retention policy" per object via the annotation APIs (ListObjectAnnotations / GetObjectAnnotation), or across a bucket via an S3 Metadata annotation table — an Apache Iceberg table in S3 Tables queryable with Athena. No object reads, no separate governance DB. (concepts/least-privileged-access)
Context stays readable in any storage class. Annotations on an object archived in S3 Glacier Flexible Retrieval or Deep Archive are directly readable without restoring the object first. You pay S3 Standard rates for annotation storage regardless of the parent object's storage class — an annotation on a Deep Archive object still costs Standard-tier storage. (concepts/storage-media-tiering)
Deletion is scoped to the object version, and the failure mode is silent. An annotation disappears only when its object version disappears. In a versioned bucket a DELETE without a version ID creates a delete marker and leaves the underlying version (and its PII annotations) in place — the object vanishes from ListObjectsV2 but is still in ListObjectVersions. A plain delete is therefore not a valid right-to-be-forgotten path; you must issue a version-targeted delete-object --version-id or expire noncurrent versions via an S3 Lifecycle rule. Lifecycle rules expire on a time schedule and cannot read annotation values — value-triggered deletion requires your own automation that reads the annotation and issues the delete. (concepts/audit-trail, concepts/immutable-object-storage)
Object Lock freezes annotations → sequencing is a design rule. While a version is under active Object Lock retention, PutObjectAnnotation/DeleteObjectAnnotation are denied (AccessDenied, "object protected by object lock retention") in both Governance and Compliance mode; reads still work. Therefore annotate the object first, then apply retention. Governance-mode version-targeted delete of a locked version needs s3:BypassGovernanceRetention + --bypass-governance-retention (gate it with approval); Compliance mode blocks deletion for everyone including root until retention expires. Event holds add a third case: releasing a hold starts the retention period rather than ending protection, so releasing it moves a deletion request further away.
Multipart copies drop annotations unless you opt in — behavior changes with object size, not command. CopyObject (≤5 GiB) copies annotations by default, but larger objects copied via multipart upload (what the CLI/SDK transfer managers use above ~8 MB) silently drop annotations with no error. Fix: standardize on --copy-props all (or the x-amz-annotation-directive: COPY header on direct CopyObject; EXCLUDE for a deliberate reduced-sensitivity copy) in every annotation-dependent copy path, and test with an object larger than the multipart threshold because a small-object test passes while hiding the bug. With the opt-in, the SDK writes annotations after the copy, so the destination briefly exists without all its annotations.
Annotation deletion and overwrite are irreversible — no version history, even in a versioned bucket. There is no delete marker and no version history for an annotation; deleting one is permanent, and overwriting an existing annotation name replaces the previous payload in place, making an accidental overwrite as destructive as a delete. Treat deletion as a gated administrative action and subscribe to s3:ObjectAnnotation:Put/:Delete event notifications (or the S3 Metadata journal table) to keep an audit trail of annotation changes. (patterns/conditional-write — the scheduled-update path guards with a conditional write on the object's ETag via --object-if-match.)
SSE-C objects can't have annotations. S3 encrypts annotations with the parent object's configuration — SSE-S3, SSE-KMS, DSSE-KMS all support annotations; SSE-C does not (InvalidRequest: "Adding metadata to object with CUSTOMER_MANAGED encryption is not supported"). This aligns with the April 2026 move to disable SSE-C by default for new buckets; a governed bucket built on SSE-C must move to SSE-S3/SSE-KMS/DSSE-KMS. On SSE-KMS, annotations use the object's KMS key, so consumer roles need kms:Decrypt and the enrichment role also needs kms:GenerateDataKey.
Read, write, and delete access are separately scoped IAM actions. Distinct actions let an enrichment process write while consumers only read: write role gets s3:PutObjectAnnotation + s3:GetObjectAnnotation/s3:GetObjectVersionAnnotation; consumer role gets the read trio + s3:ListObjectAnnotations; s3:DeleteObjectAnnotation is restricted to a dedicated admin role, with an explicit Deny on delete added to the enrichment/consumer policies for defense in depth (deny overrides any allow reaching the role through another policy). The deny covers deletion only — the enrichment role can still overwrite governance-context, so the audit trail remains the detection mechanism. (concepts/attribute-based-access-control, concepts/defense-in-depth) Replication needs its own pair: s3:GetObjectVersionAnnotationForReplication on source + s3:ReplicateObjectAnnotation on destination; annotations replicate independently of and after the object (treat as eventually consistent; verify via the annotation table's per-annotation replication_status column; deny s3:ReplicateObjectAnnotation in the replication role to hold annotations in the source Region while objects replicate).
Generated labels are proposals, not decisions. S3 stores whatever label you write and never checks its correctness, so a wrong generated PII flag or retention status that drives automated expiration becomes a wrong deletion. Keep a validation boundary between authoring context and acting on it: the payload's review_required field starts true and is set false only after validation, so downstream automation keys off a reviewed label; generated_by/generated_at give the audit trail. Treat the enrichment agent as a proposer, never an approver — for irreversible/high-impact actions (deletion, expiring retention, cross-border transfer) require explicit human approval regardless of model confidence. (concepts/audit-trail)

Reference architecture (event-driven, four paths)¶

S3 sits at the center of an event-driven system with four paths: a write path enriches new objects, a maintenance path keeps context current, a backfill path annotates pre-existing objects, a read path serves consumers from annotation context alone.

Producer PutObject → governed S3 bucket.
S3 emits an object-created event to EventBridge (chosen over direct invocation so you can filter on prefix/size and route the same event to multiple targets; a direct notification or an SQS queue is a reasonable simpler choice if you don't need that routing).
EventBridge invokes the enrichment Lambda.
The function calls an Amazon Bedrock enrichment agent to classify the object and extract retention/jurisdiction/PII context; the function validates that the output parses as JSON and conforms to the schema before writing (models often wrap JSON in a Markdown code fence; the annotation payload must be the raw document).
The function writes context with PutObjectAnnotation; the write emits an s3:ObjectAnnotation:Put event downstream consumers can subscribe to.
On a schedule, EventBridge Scheduler re-invokes the enrichment function to re-assess and update annotations in place — guarded by a conditional write on the object's ETag (--object-if-match) so a scheduled update can't overwrite context belonging to a version the function never evaluated.
S3 Batch Operations invokes a backfill Lambda for objects that predate the pipeline.
The backfill function writes context with PutObjectAnnotation.
A consumer asks a Bedrock consumer agent a governance question.
The agent lists/retrieves relevant annotations with ListObjectAnnotations + GetObjectAnnotation (annotation context only, never the object body).
For cross-object queries the agent queries Athena.
Athena reads the S3 Metadata annotation table (Iceberg on S3 Tables) that S3 maintains automatically from the object annotations.

Operational numbers and constraints¶

Capacity: up to 1,000 annotations per object version; payload 1 byte – 1 MB; name 1–512 bytes, letters/digits/underscore/period/hyphen only, and cannot begin with the reserved aws or s3 prefixes. Payloads must be valid UTF-8 — Base64-encode any binary content. Version the payload schema so consumers can evolve.
S3 Metadata annotation table: populated asynchronously; initial backfill minutes-to-hours depending on object/annotation count; results reflect the most recent refresh (~1 hour typical) → eventually consistent, not real time.
Cost: you pay S3 Standard storage rates for annotation payloads regardless of the parent object's storage class; request charges for PutObjectAnnotation and reads; optional annotation-table storage + request + Athena scan costs. Each annotation is copied/replicated as a separate PUT, so bulk-copy/replication of objects carrying many annotations incurs request cost proportional to the annotation count. Confirm current pricing on the S3 pricing page (Annotations under Management & Insights) + S3 Metadata/Athena pages.
Example governance-context payload fields: schema_version, classification, retention{policy, effective_date, expires_on, status}, jurisdiction, data_residency, pii{present, categories[]}, rationale, generated_by, generated_at, review_required. This post uses a single annotation named governance-context (authored/consumed as a unit, internally consistent, one op per read/write) rather than granular annotations (which allow per-field updates, field-scoped overwrite blast radius, and finer-grained table queries) — distributing a record across annotations is a design choice.

Annotations vs. other object-context mechanisms¶

The post is explicit that annotations occupy a specific niche — larger, structured, mutable context that travels with the object — and are the wrong tool when the access pattern is relational/join-heavy, when you need IAM-condition or Lifecycle filtering on the values themselves, or when there's a very high write rate on the same object.

Design decision	S3 annotations	Object tags	User-defined metadata	External data source
Max per object version	1,000	10	Fixed at upload	N/A
Max size	512-byte name + 1 MB payload	128-char key + 256-char value	≤2 KB total	Unbounded
Data format	Any UTF-8 (JSON/XML/YAML/text)	Key-value strings	Key-value strings	Any
Mutable after write	Yes	Yes	No (requires object copy)	Yes
Set during upload	No (after upload only)	Yes	Yes	N/A
IAM-condition / Lifecycle filtering	No	Yes	No	N/A
Query at scale	S3 Metadata annotation table	S3 Metadata live inventory table	S3 Metadata live inventory table	Rich / relational

One consequence: because annotations are written after the object exists, there is a short upload-to-enrichment window during which the object carries no governance context. If controls can't tolerate that gap, the context can arrive with the upload via user-defined metadata or a pre-registration step.

Caveats¶

Reference/guidance post, not a production retrospective. Every payload, IAM statement, and CLI call is illustrative; there are no measured percentiles, throughput, or cost figures beyond the pricing-model structure. The annotation-table refresh ("typically within about an hour") and backfill ("minutes to hours") are the only timing figures, and both are qualitative.
The SSE-C default-disable date (April 2026) and the behaviors described are documented S3 rules at time of writing; confirm against current S3 documentation before building.

Source¶

systems/aws-s3 — S3 annotations are a feature of S3; this post is the canonical wiki disclosure of the annotation API + its version-lifecycle semantics.
systems/s3-tables / systems/apache-iceberg — the S3 Metadata annotation table is an Iceberg table in S3 Tables.
concepts/data-residency — a residency/jurisdiction label that travels with the object (and its DR copy) is one of the five annotation properties.
concepts/audit-trail — annotation-change event notifications / the S3 Metadata journal table as the only record of a governance decision once an annotation is overwritten or deleted.
patterns/conditional-write — the ETag-guarded scheduled-update path (--object-if-match).