CONCEPT Cited by 1 source
User-agent rotation¶
Definition¶
User-agent rotation is the practice of switching the
User-Agent request header across requests (or in response to
origin-side blocks) to evade user-agent-based allow/deny rules.
A crawler whose declared UA is blocked might cycle through a
pool of generic browser UAs — Chrome/124 macOS,
Firefox/127 Linux, Safari/17 iPhone — so that no single
origin-visible identifier attracts sustained enforcement.
Because the User-Agent header is entirely client-controlled
and unauthenticated, rotation defeats any allow/deny policy
whose selector is UA string. The only durable countermeasure is
to move identity off the UA string entirely — cryptographic
primitives (Web Bot Auth) or
content-independent signals
(ML fingerprinting).
Canonical instance¶
Perplexity AI's stealth crawler (Cloudflare, 2025-08-04)
uses a single impersonated UA (Chrome/124.0.0.0 on macOS), but
combines UA spoofing with IP
rotation + ASN rotation in response
to enforcement. The post documents the escalation pattern:
declared UA blocked → switch to stealth UA + rotate IPs; stealth
UA blocked on some IPs → rotate to different IPs / ASNs. See
patterns/stealth-on-block-fallback.
Seen in¶
- sources/2025-08-04-cloudflare-perplexity-stealth-undeclared-crawlers — canonical wiki instance; the stealth crawler's UA is a single impersonated Chrome string rather than a rotating pool, but the impersonation serves the same evasion purpose and composes with IP + ASN rotation.
Related¶
- concepts/stealth-crawler.
- concepts/asn-rotation / concepts/ip-rotation-as-evasion.
- concepts/ml-bot-fingerprinting.
- patterns/stealth-on-block-fallback.
- systems/web-bot-auth — cryptographic alternative to UA-based identity.