SYSTEM Cited by 1 source
S3 Batch Operations¶
S3 Batch Operations is AWS's primitive for issuing a single
action (tag, copy, invoke Lambda, restore, etc.) against a
large list of S3 objects defined in a manifest. The manifest
is itself an S3 object — typically a CSV of
bucket,key[,version] rows, or the output of an
S3 Inventory report.
Role for this wiki¶
First-class canonical use case: scalable per-object tagging
for tag-based lifecycle expiration. Per the AWS constraint,
S3 Batch Operations does not support Delete as an action;
deleting millions of objects via per-object DELETE API calls
doesn't scale even with the batch-delete API. The canonical
workaround is
tag-then-
expire: use Batch Operations PutObjectTagging to tag the
objects, then let the bucket's lifecycle policy expire tagged
objects.
Known cost and manifest gotchas (from Yelp's 2025-09-26 post)¶
- Flat $0.25 per bucket per job. Becomes the dominant cost
for buckets with low object-change volume. Yelp's dispatch
rule: use Batch Operations only for high-volume buckets;
directly tag objects (per-object
PutObjectTaggingAPI) for most buckets "that generate low order access logs." (Source: sources/2025-09-26-yelp-s3-server-access-logs-at-scale) - Header rows in manifests cause job failures. Athena query results include a header row that S3 Batch Operations interprets as a bucket name — "causing job failures." Yelp regenerates manifest files in memory without headers.
- Object keys must be URL-encoded to
quote_plus(key, safe="/")equivalence before being written into a manifest. This is the canonical documented input contract — AWS docs.
Seen in¶
- sources/2025-09-26-yelp-s3-server-access-logs-at-scale —
canonical wiki use case. Yelp runs S3 Batch Operations
PutObjectTaggingjobs to mark compacted source SAL objects for lifecycle expiration; the $0.25 per-bucket flat fee drives Yelp's per-bucket dispatch rule (high-volume → Batch Ops, low-volume → direct per-object tagging). Also discloses the no-delete-action constraint that forces the tag-and- expire pattern and the manifest-header + URL-encoding gotchas.
Related¶
- systems/aws-s3 — parent service.
- systems/s3-inventory — one source of Batch Operations manifests.
- patterns/object-tagging-for-lifecycle-expiration — the canonical deletion-at-scale pattern built on Batch Operations + lifecycle policies.
- patterns/s3-access-based-retention — higher-level Yelp retention scheme that Batch Operations executes the tagging step of.
- concepts/url-encoding-idiosyncrasy-s3-keys — the downstream hazard when keys in SAL need to be translated to Batch Operations manifest entries.