CONCEPT Cited by 1 source

Shared test environment contention¶

Definition¶

Shared test environment contention is the failure mode where a single test/staging environment is the serialisation point for multiple in-flight changes — so teams can't validate their changes in parallel without interfering with each other, and are forced either to (a) accept cross-contamination or (b) serialise testing and eat the delay.

The textbook setup this describes¶

From sources/2022-06-09-zalando-accelerate-testing-in-apache-airflow-through-dag-versioning:

"Only one version of an Airflow DAG such as our marketing ROI pipeline can exist in each environment. When multiple features are developed at the same time, they have to share the test environment which oftentimes leads to conflicts since testing in isolation is not possible. Alternatively, the features can be tested sequentially which leads to delays."

Classic shape: two servers (prod + test), one DAG version per env, N teams developing in parallel, each wanting to validate their change end-to-end against realistic-size data.

Why it's worse for pipelines than for services¶

For request/response services, a shared staging env can often absorb many parallel branches with per-feature toggles. For batch pipelines with no ground truth — like Zalando's marketing ROI pipeline — validating a change requires running the whole pipeline end-to-end on realistic data, observing the output, and comparing to assumptions. Two features' code cannot both write the same table at the same time without corrupting each other's answer. The comparison step is the point; cross-contamination breaks it.

Resolutions¶

Three practical ways out:

Per-server-per-PR. Separate orchestrator server per PR. Correct, but ~30 min + $$$ (e.g. MWAA spin-up cost).
Per-PR ephemeral environments on shared infra — the Zalando approach. One server, many named pipeline environments + data environments, each per-PR ephemeral. <1 min creation; cleanup via cron.
Sequential testing — the default if you do nothing. Doesn't scale past ~2 in-flight features before developer velocity collapses.

The wiki's canonical (2) instance is patterns/per-pr-airflow-environment-via-dag-versioning.

Seen in¶

sources/2022-06-09-zalando-accelerate-testing-in-apache-airflow-through-dag-versioning — the problem-statement framing for Zalando's marketing ROI pipeline.