Skip to content

SYSTEM Cited by 1 source

Bulk Executor for DynamoDB

Bulk Executor is an open-source (awslabs/amazon-dynamodb-tools) command-line tool that runs actions in bulk against many or all items in an Amazon DynamoDB table — e.g. mass-delete old items, count with a filter, copy a table, compare (diff) two tables for equality, or mass-load synthetic data. Its design goal is that running a bulk command should feel like a local script (real-time output, interruptible with ^C) while transparently scaling to hundreds of parallel workers without the user managing any infrastructure.

Architecture

Three requirements shaped most decisions: (1) feel like a local script; (2) scale to hundreds of parallel workers with no infra management; (3) make it easy to add new commands without tight framework coupling.

  • Executor: AWS Glue (serverless Spark). A thin local Python client starts a Glue job; Glue fans the work across workers. Glue was chosen over SQS+Lambda, Fargate, EMR, and Ray because it provides both serverless parallelism and a built-in DynamoDB connector, so much of the hard work came "at no cost." (Source: sources/2026-10-13-aws-bulk-executor-for-amazon-dynamodb-part-2-lessons-we-learned)
  • Two data paths — connector vs. direct API:
    • Connector path (find, count): reads DynamoDB into a Spark DataFrame, giving where (Spark-SQL condition), orderBy, and limit almost for free. The connector cannot delete items and is type-lossy (no string-set vs. list-of-strings distinction).
    • Direct-API path (delete, fill, copy, diff, load-export, scancount): bypasses the connector, iterating partitions with foreachPartition and calling DynamoDB directly (batch_writer() for writes, segmented scan for scancount). Chosen for type fidelity (diff/copy) or speed (scancount).
  • scancount uses a DynamoDB segmented scan (Segment/TotalSegments) with Select: COUNT and a pushed-down FilterExpression, so nothing leaves the database except the count — no items are hydrated into Spark to be filtered and discarded. Consistently faster than connector-based count; the tradeoff is a raw --filter-expression vs. the higher-level --where.
  • Output overflow to S3: large find/diff results are written to an S3 bucket; only the first ~10 rows plus a link stream to the terminal. Each diff task writes its own S3 object rather than centralizing output back to the driver (which wouldn't scale).

Operational lessons (the "hard way")

  • ^C propagation. ^C kills the local client but leaves the Glue job running to its timeout. An interrupt handler sets an interrupted flag that a continuously-running watcher loop (normally watching Glue state) acts on to stop the Glue job before a clean local exit.
  • CloudWatch Live Tail replaced slow log polling for near-real-time output, but it samples above 500 events/sec, delivers out of order, and lags job completion by seconds. Mitigations: S3 overflow + 10-row preview; a reordering layer; a "microwave popcorn monitor" that stops listening only after a few seconds of silence.
  • Four-layer error handling to avoid flooding the terminal with hundreds of identical stack traces: client-side param validation; a Spark error_accumulator that reports only the first error then shuts down gracefully; a Live-Tail pattern matcher that stops the job early on never-self-resolving errors (AccessDeniedException, ModuleNotFoundError, OutOfMemoryError); and an app-specific exception type that renders a plain message (no stack trace) for user errors — only developer errors get traces.
  • Distributed rate limiter coordinates across all workers for the direct-API commands (the connector rate-limits itself; direct calls don't). Controlled by --XMaxReadRate/--XMaxWriteRate, inferred from account/table quotas if unset. Warns if the max rate can't finish within --XTimeout (default 1 h).

Cost estimation

  • Reads: table-metadata size × RRUs for a full scan × per-RRU price from the Pricing API. ⚠ Table-size metadata refreshes only ~every 6 hours — a freshly-loaded table reads smaller than it is.
  • Provisioned mode: heuristic of two-thirds of on-demand (÷1.5), deliberately biased to over-quote.
  • delete: two-phase — read cost up front, write cost after the scan finds the match set. fill: generate sample items to learn size, then price.
  • Glue DPU cost is deliberately not estimated (hard to predict; small vs. DynamoDB cost at scale); --XWaitForDPU reports actual DPU cost afterward.
  • Marked up-front cost prediction so a user can abort before launching an expensive scan.

Safety: mandated PITR

Any table receiving a mutating command must have point-in-time recovery enabled — a guardrail so a wrong --where clause is recoverable. The revert-export command undoes writes in place from a PITR-driven incremental export.

Extensibility

Each command is a Python module loaded by name; the only contract is a run(job, spark_context, glue_context, parsed_args) function plus shared conventions (error_accumulator, the distributed rate limiter, the cost-estimate library). This loose coupling attracted outside contributions (diff, sql, load, load-export, revert-export). Every --X… flag parameterizes the harness itself (vs. the command), modeled on the JVM convention. An "AI linter" (a folder of markdown rule files) checks invariants that are hard to unit-test — chiefly that three differently-formatted statements of the required IAM permissions (README prose, role-creation logic, IAM-policy-simulator verification) do not drift apart.

Seen in

  • sources/2026-10-13-aws-bulk-executor-for-amazon-dynamodb-part-2-lessons-we-learned — build retrospective (Part 2): Glue-as-executor choice, connector-vs-direct data paths, scancount segmented scan, cost estimation (÷1.5 heuristic), mandated PITR, ^C propagation, Live Tail sampling, four-layer error handling, distributed rate limiter, plugin model, AI-assisted Glue 4→5 testing + AI linter. (Source: this article)
Last updated · 854 distilled / 2,448 read