Skip to content

CONCEPT Cited by 2 sources

Automated vs manual testing complementarity

Definition

Automated vs manual testing complementarity is the framing that automated testing is a layer in a broader testing strategy, not a substitute for manual testing. Specifically for accessibility: automated tools catch only the subset of violations that are machine-decidable from the rendered DOM. Screen-reader UX, cognitive load, alt-text meaningfulness, keyboard navigation feel, and assistive-technology compatibility require human judgment — often from users with disabilities themselves.

Shape of the boundary

What automated a11y tooling (e.g. Axe) catches:

  • Missing alt attributes on images.
  • Colour-contrast ratios below WCAG thresholds.
  • Missing form labels.
  • Invalid ARIA attribute / role combinations.
  • Structural issues detectable via DOM inspection.

What automated a11y tooling cannot catch:

  • Whether the alt text is meaningful (alt="image" passes an automated check but fails actual accessibility).
  • Whether heading structure reflects document semantics.
  • Whether focus order matches visual reading order.
  • Whether screen-reader pronunciation is comprehensible (this varies by screen reader, language, and user familiarity).
  • Whether the overall flow is navigable by keyboard in practice.
  • Whether the cognitive load is reasonable for users with cognitive disabilities.
  • Whether design-system choices genuinely serve assistive-tech users.

Plus automated tools sometimes false-positive against design conventions: an Axe rule flags a pattern the team has deliberately chosen, and the flag is wrong in that context — see patterns/exclusion-list-for-known-issues-and-out-of-scope-rules for the management lever.

The layered strategy

Slack's explicit layering (canonical example):

  1. Involve people with disabilities early in design — before anything is coded.
  2. Conduct design and prototype reviews with those users — before implementation locks in.
  3. Manual testing across all supported assistive technologies — ongoing, per feature.
  4. Automated Axe checks — continuous, scaled across many pages and flows, catching the machine-decidable subset.
  5. External manual accessibility testers — periodic third- party audits for coverage and independence.

Verbatim: "Automated tools can overlook nuanced accessibility issues that require human judgment, such as screen reader usability. Additionally, these tools can also flag issues that don't align with the product's specific design considerations."

Generalisation

The same complementarity applies to other test domains:

  • Security testing — static analysis + SAST catches pattern vulns; pen-testing catches business-logic vulns.
  • Performance testing — automated load tests catch throughput ceilings; observability catches emergent degradation.
  • Visual regression — screenshot diffs catch structural visual changes; human review catches "does this still look right" judgment.

The canonical anti-pattern is treating automation as sufficient — building a green-CI culture that equates passing automated checks with the quality they are supposed to approximate.

Seen in

  • sources/2025-01-07-slack-automated-accessibility-testing-at-slack — Slack explicitly frames automated a11y testing as a layer, not a replacement. "At Slack, we see automated accessibility testing as a valuable addition to our broader testing strategy. This broader strategy also includes involving people with disabilities early in the design process, conducting design and prototype review with these users, and performing manual testing across all of the assistive technologies we support."
  • sources/2025-11-19-slack-android-vpat-journey — the third-party / periodic / manual layer at Slack: a 2024 VPAT audit by an outside vendor. Surfaced 8 recurring accessibility themes on Android that the automated Axe-in- Playwright CI suite alone would not have flagged — some because they required screen-reader context (error-not-announced), some because they required judgment about platform convention (top-app-bar-as-heading; strikethrough announcement), some because they required dexterity / gesture considerations (drag-and-drop). Composes with the 2025-01-07 ingest: the two Slack posts canonicalise the automated and manual layers of Slack's same broader a11y strategy.
Last updated · 470 distilled / 1,213 read