Skip to content

AWS 2026-10-13 Tier 1

Read original ↗

Bulk Executor for Amazon DynamoDB, Part 2: Lessons we learned the hard way

Summary

Part 2 of the AWS Architecture Blog's write-up of Bulk Executor, an open-source (awslabs/amazon-dynamodb-tools) command-line tool that runs actions in bulk against many or all items in a DynamoDB table (mass delete, count, copy, diff, fill, load-export). Where Part 1 covered purpose and usage, Part 2 is a build retrospective: the design tradeoffs, surprises, and workarounds the team hit putting hundreds of parallel workers behind a tool that should "feel like a local script." The engine is AWS Glue (serverless Spark), chosen for serverless parallelism plus a built-in DynamoDB connector. Most of the hard-won lessons are not big architectural choices but a pile of small frustrations that each made the tool unusable until fixed — ^C not reaching the remote Glue job, CloudWatch Live Tail sampling away results, one bad --where clause producing hundreds of identical stack traces, unbounded read/write rates triggering throttles, and the inability to predict cost before launching an expensive scan.

Key takeaways

  • AWS Glue chosen as the serverless parallel executor over SQS+Lambda, Fargate, EMR, and Ray — because it gave both serverless worker fan-out and a built-in DynamoDB connector, so the DataFrame where/orderBy/limit primitives came "for free." The first command (find, with --where, --orderby, --limit) required almost no code; count and delete reuse the same find path. (Source: this article)
  • The Glue DynamoDB connector doesn't fully represent DynamoDB types — it simplifies the data (e.g. no distinction between a string set and a list of strings) and cannot delete items at all. So delete, fill, copy, diff, load-export, and scancount bypass the connector and make direct DynamoDB API calls (batch_writer() / segmented scan) via foreachPartition. For diff/copy the motive is type fidelity; for scancount it is speed. (Source: this article)
  • scancount (direct segmented scan) is consistently faster than connector-based count because of less overhead: with Select: COUNT + a pushed-down FilterExpression, nothing leaves the database except the count — no items are hydrated into the Glue/Spark context just to be filtered and discarded. Each worker scans its own Segment of TotalSegments in parallel. Tradeoff: scancount takes a raw --filter-expression instead of the higher-level --where Spark-SQL syntax (no DataFrame involved). (Source: this article)
  • Cost estimation before launch is a first-class feature. For reads, the tool reads table metadata size (⚠ refreshed only ~every 6 hours — a recently-loaded table can read as smaller than it is), computes read request units for a full scan, and multiplies by the per-RRU price from the Pricing API. For provisioned tables (where cost depends on provisioned level, autoscaling headroom, job duration) it uses a heuristic: two-thirds of the on-demand price (÷1.5) — reasoning that a table at 100% utilization costs ~30% of equivalent on-demand but 100% is rarely achievable, ~50% utilization is ~60%, so 67% is a deliberately slightly-high quote. (Source: this article)
  • delete prices in two phases (read cost up front, write cost only after scanning finds the matching set, since match count is unknown until then); fill generates sample items first to learn item size, then prices. Glue DPU cost is deliberately not estimated — hard to predict, and for large jobs it's a small fraction of the DynamoDB cost anyway (a cheap Glue worker drives huge DB activity); --XWaitForDPU reports actual DPU cost after the fact. (Source: this article)
  • Point-in-time recovery (PITR) is mandated on any table receiving a mutating command (delete, etc.) — a deliberate guardrail so a wrong --where clause is recoverable. This motivated a revert-export command that undoes writes in place from a PITR-driven incremental export. (Source: this article)
  • ^C must be propagated to the remote job manually. Pressing ^C kills the local Python client but leaves the Glue job running to its timeout. Fix: an interrupt handler sets an interrupted flag that an always-running watcher loop (normally watching Glue state) notices and uses to actively stop the Glue job before a clean client exit — so one ^C stops both sides. (Source: this article)
  • CloudWatch Live Tail replaced log polling for near-real-time output — but it samples. Live Tail displays at most 500 events/second; beyond that it streams only a sample, so large find/diff result sets came back incomplete. Fix: write large output to an S3 bucket and stream only the first 10 results to the terminal plus a link to the full dataset. Two more Live Tail surprises: events can arrive out of order (needed a reordering layer), and Glue jobs can finish before their final log events propagate (seconds of delay) — so a "microwave popcorn monitor" stops listening only after a few seconds of silence, not the instant the job ends. (Source: this article)
  • Four-layer error handling to avoid "scary" output: (1) client-side param validation before Glue even starts (catch a typo locally); (2) a Spark error_accumulator so each task adds its exception instead of propagating — the driver prints only the first error, one clean message instead of hundreds of duplicates, then graceful shutdown; (3) a client-side pattern matcher on the Live Tail stream for never-self-resolving errors (AccessDeniedException, ModuleNotFoundError, OutOfMemoryError) that triggers an immediate job stop; (4) an app-specific exception type a command throws to show a plain error (no stack trace) for user errors — only developer errors get stack traces. Plus color-coding (red/yellow) and a final success/fail line. (Source: this article)
  • A distributed rate limiter coordinates across all Glue workers for the direct-API commands (the connector has built-in rate limiting; the direct-call commands don't, so unconstrained worker loops cause throttles). Controlled by --XMaxReadRate / --XMaxWriteRate; if unset, the tool infers rates from account/table settings and quotas. A recent addition warns (yellow) if the max rate can't finish the work inside --XTimeout (default 1 hour) — fail-fast so the user raises the rate or timeout before waiting. (Source: this article)
  • Loose-coupling plugin model paid off in outside contributions. Each command is a Python module loaded by name; the contract is just a run(job, spark_context, glue_context, parsed_args) function plus shared conventions (the error_accumulator, the distributed rate limiter, the cost-estimate library). diff, sql, load, load-export, revert-export were contributed by people outside the core team. Every --X… flag is a parameter to the harness itself (vs. the command), modeled on the JVM convention. (Source: this article)
  • AI used for the hard-to-run and hard-to-test parts. The AWS Glue 4→5 migration (faster job startup, new connector that loads directly to a DataFrame without the old DynamicFrame intermediate step) was de-risked by having AI generate and run a multi-hour stress-test suite comparing old vs. new connector across table sizes and read/write patterns, then produce a comparison report — turning a long-postponed backlog item into one working session. They also built an "AI linter": a folder of markdown rule files describing invariants hard to unit-test — e.g. keeping three differently-formatted statements of the required IAM permissions (English README prose, role-creation logic, IAM-policy-simulator verification logic) from drifting apart; "every command must impose rate limiting"; "every command with large output must write to S3 with a bounded terminal preview + link." AI runs the linting on request; AI authored the rule files too. Later upgraded to Glue 6 for ~30% lower DPU-hour cost. (Source: this article)

Systems / concepts extracted

  • systems/bulk-executor (NEW page — primary subject) — open-source Glue-backed CLI for bulk operations on DynamoDB tables.
  • systems/aws-glue — serverless Spark runtime chosen as the parallel executor; its DynamoDB connector is both the enabler (DataFrame primitives) and a constraint (type-lossy, can't delete). Glue 4→5→6 migration path.
  • systems/dynamodb — target data store; segmented scan with Select: COUNT + pushed-down FilterExpression, batch_writer(), PITR requirement for mutations.
  • systems/apache-spark — DataFrame where/orderBy/limit, foreachPartition for distributing direct deletes, and the Spark accumulator as a single-error aggregation channel.
  • systems/aws-cloudwatch — Live Tail for near-real-time job output, with its 500-events/sec sampling cap, out-of-order delivery, and propagation-delay-after-job-completion gotchas.
  • systems/aws-s3 — overflow sink for large find/diff output; each diff task writes its own S3 object (no centralizing back to the driver).
  • concepts/point-in-time-recovery — mandated as a precondition for mutating commands; the undo substrate for revert-export.
  • concepts/predicate-pushdown — scancount pushes the filter + COUNT into DynamoDB so only the count returns.
  • concepts/scatter-gather-query — DynamoDB segmented/parallel scan across workers, each owning one segment.
  • concepts/performance-prediction — up-front cost estimation; the ÷1.5 provisioned-mode heuristic; deliberately not modeling DPU.
  • Rate limiting across distributed workers, a plugin/extensibility contract, the AI-linter-for-triplicated-invariants idea, and the ^C-watcher-flag pattern are recorded as prose/tags per the taxonomy gate (one-off or single-source).

Caveats / notes

  • Demonstration-and-tool post, not a production-scale retrospective — there are no throughput/latency/dollar numbers for a specific workload; the claims are qualitative ("consistently faster," "small fraction of cost," "~30% lower").
  • The ÷1.5 provisioned-cost heuristic is an admitted approximation and is deliberately biased to over-quote.
  • The 6-hour staleness of DynamoDB table-size metadata is an explicit footgun for cost estimates on freshly-loaded tables.

Source

Last updated · 854 distilled / 2,448 read