Skip to content

CLOUDFLARE 2026-10-30 Tier 1

Read original ↗

4 real? Tracking exploitation and patching of eMudhra's weak keys

Summary

This is Cloudflare's follow-up, ten days later, to its first eMudhra post. On 2026-10-19 Cloudflare learned that eight eMudhra intermediate CA keys were absurdly weak: the "secret" values were the numbers 4, 56, 66, 115, 155, 156, 230 and 237, where a 115-digit value was expected. Anyone can recover these signing keys from the public certificates, so anyone can mint a certificate for any domain that an unpatched client (one that does not enforce Certificate Transparency) will accept. That makes this incident different from earlier CA compromises (DigiNotar, TÜRKTRUST, TrustWave), where only a small set of actors had the key. The post reports three things. (1) Patching Cloudflare: the origin-facing Pingora proxies were patched in hours, and an AI-assisted analysis of the source code found every other system that trusts the weak keys within four hours. Each system got a ticket with a preliminary analysis, and the remediation deadline depended on the impact. (2) Exploitation telemetry: Pingora now logs every outbound handshake that presents a weak-key chain. In one week it saw about 22,000 such handshakes. All were refused where verification was on. About 160 carried forged certificates for unrelated domains, one dated less than an hour after the Mozilla bug became public. There was no evidence of interception. (3) Visitor patch adoption: Cloudflare used Challenge Pages to send two background probes, one to a weak-chain endpoint and one to a control endpoint. Over 50 million probes, up-to-date browsers are mostly patched, but bots, old browsers, and embedded browsers (Android WebView, 30–40% still accept) lag. Across large networks, more than half of probes still accept the compromised chain, and the trend has been flat since the first weekend. The forward argument repeats: a trust store must be a fast-update channel for revocation and CT keys (upki), not a static root list.

Key takeaways

  1. Key recoverability changes the threat model from "few attackers" to "anyone". Past CA compromises gave the key to a contained set of actors. Here the intermediate private keys "could be recovered almost instantly by anyone from the public certificates themselves." This is the break-once, forge-everywhere property with an unbounded attacker set — the blast radius is every unpatched client that does not enforce CT. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  2. Patch the highest-traffic trust path first. The origin-facing Pingora proxies carry "most of our outbound connections", so they were patched first, within hours, to reject the weak eMudhra keys. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  3. AI-assisted trust-dependency inventory, then per-system tickets. At the same time, Cloudflare ran an AI-assisted analysis of its source code to find all systems that trust the weak keys. The full analysis took four hours. Each system got a ticket with a preliminary analysis. The owning team had to confirm or correct it the same day. High-impact systems had to patch immediately, and lower-impact ones by the end of the following week. Humans verify the machine's triage; the machine supplies breadth. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  4. Put detection logging on the choke point. Because Pingora terminates the "vast majority of outgoing TLS connections from a Cloudflare machine", one log statement there covers almost all attempts to snoop on Cloudflare's connections with the weak keys. The rollout finished on 2026-10-21. This is observability placed where the traffic concentrates, not on every service. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  5. Most weak-key handshakes were tests or self-signed setups — but forging started within the hour. Of about 22,000 handshakes: two-thirds were fetches of test sites (for example emudhra.we-have-a-problem.com), and almost all of the rest went to origins that had issued themselves a certificate under the weak intermediate. About 160 had SANs for unrelated organizations' domains (some consumer IoT), were created after the leak, and some were valid for more than two years (longer than any public CA may issue). One was dated 2026-10-20 18:19 UTC, less than an hour after Mozilla's bug became public. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  6. Use hostname mismatch and multi-vantage consistency to tell "server default cert" from "on-path interception". The forged certificates did not match the SNI, so hostname verification would reject them anyway. The same certificate came back to connections from many Cloudflare locations, so it came from the server itself and not from one network path. Conclusion: no evidence of interception, but this cannot exclude carefully targeted attacks; for those, Cloudflare relies on the usual indirect compromise signals. Every handshake configured to verify was refused (fail-closed). (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  7. Measure client trust with treatment/control probes on Challenge Pages. Cloudflare cannot test a visitor's trust store without breaking the connection. So, on a fraction of challenge pages, it sent two probes: ca1.cloudflarecaptcha.com (eMudhra-signed chain) and ca2.cloudflarecaptcha.com (normal chain, the control). The success or failure of each probe shows whether that client still trusts the weak keys. This reuses the Challenge Page RUM substrate that Cloudflare already uses for network-speed probes. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  8. Iterate the experiment, and log every change — each one moves the curve. The probe setup changed five times in one week: the control was converted to an eMudhra chain without SCTs (faster than shipping a new endpoint through the slow challenge-platform release), an unrelated platform rollback caused a gap from 10-23 to 10-27, the missing G3 cross-sign was added and SCTs removed, ca2 was re-chained to the AAA Certificate Services root, and sampling went from 1% to 10% of challenge pages on non-paying zones. A later bot-detection release (10-29) reclassified traffic that claimed to be Chrome 150–154 on macOS. That looked like patching but was a classification change. Without the change log, these steps would read as real trend changes. (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  9. The patch-adoption picture: browsers fine, long tail not. Modern browsers are mostly patched. Bots, old browsers, and embedded browsers are not: Android WebView still accepts at 30–40%. Across large ASNs, more than half of probes accept the compromised chain, both with and without bot signals. There was clear patching over the first weekend and little movement after. Chrome 150–154 on Linux is the only cohort with a continuing decline. "Bots are also just software that need security updates" — so if they are representative, "the picture is quite dire." (Source: sources/2026-10-30-cloudflare-emudhra-patching)
  10. Revocation reach is the real bottleneck → updatable trust stores. The data supports the first post's argument: distrust reaches browsers fast, but not the long tail. Cloudflare has a quick-revocation mechanism in some products and wants it in more systems and in the industry, via projects like upki, which treat a trust store as quickly updatable with revocation and CT keys (concepts/certificate-revocation). eMudhra signed a CRL first with what may have been a correctly generated key and then re-signed it with the weak key; that should have set off alarms. (Source: sources/2026-10-30-cloudflare-emudhra-patching)

Operational numbers & facts

  • 8 weak intermediate keys; the "secret" values were 4, 56, 66, 115, 155, 156, 230, 237 (a 115-digit value was expected).
  • 2026-10-19 — Cloudflare learns of the issue; Pingora origin-facing proxies patched within hours.
  • 4 hours — AI-assisted source-code analysis covering all systems that trust the weak keys; owner teams confirm or correct the same day; high impact → patch now, low impact → end of the following week.
  • 2026-10-20 18:19 UTC — earliest forged certificate seen, < 1 h after Mozilla's bug report became public.
  • 2026-10-21 — Pingora weak-key handshake logging fully rolled out.
  • ~22,000 weak-key handshakes in one week: ~⅔ test sites, most of the rest self-configured origin certificates, ~160 forged certificates with unrelated SANs (some valid > 2 years). 0 accepted where verification was on.
  • 50 million challenge-page probes in one week; sampling raised 1% → 10% of challenge pages on non-paying zones (2026-10-27 18:18 UTC).
  • Probe change log (UTC): 10-23 15:30 control → eMudhra chain without SCTs; 10-23 16:30 platform rollback (probes return 10-27 10:00); 10-27 16:37 G3 cross-sign added and SCTs removed; 10-27 16:49 ca2 chained to AAA Certificate Services root; 10-29 ~18:00 bot detection reclassifies "Chrome 150–154 macOS".
  • Android WebView: 30–40% still accept. Large ASNs: > 50% accept.
  • At writing: all high-priority Cloudflare systems patched; a few confirmed low-impact tickets remain.

Systems / concepts extracted

  • Systems: Pingora (patched first; the outbound-TLS choke point that hosts the weak-key handshake logging), Turnstile / Challenge Pages (the client-trust probe surface), Bot Management (bot-signal split and the 10-29 detection that reclassified traffic), Web PKI (the trust ecosystem), Certificate Transparency (CT enforcement protects clients; SCTs as a probe variable). Listed in systems: but no page (named in passing; Systems gate): eMudhra (the CA; no architecture described), upki (updatable trust-store project under the rustls org), Android WebView (lagging embedded-browser cohort).
  • Concepts: certificate revocation (new page — now supported by two sources), chain of trust, trust anchor / trust-store distribution, blast radius, observability (choke-point logging), RUM (challenge-page probes), measurement bias (bot-skewed, JS-only sample; classification confound), fail-closed verification.
  • Single-source ideas recorded as prose/tags (NOT minted): AI-assisted trust-dependency inventory with human-confirmed per-system tickets, impact-tiered remediation deadlines, treatment-vs-control client probes to measure trust-store state without breaking connections, multi-vantage certificate consistency to rule out on-path interception, instrument the choke point. Lint candidates at ≥3 citations.

Caveats

  • The probe sample is heavily skewed: challenge-page traffic contains many (unconfirmed) bots, and only JavaScript-executing clients send probes. The numbers are directional, not a population estimate.
  • The probe setup changed several times during the week, and a platform rollback left a gap of about 3.5 days. The graphs (not reproduced here) contain jumps caused by these changes.
  • "No evidence of interception" is not proof of absence: targeted interception would not appear in Pingora's logs unless aimed at Cloudflare's own connections.
  • The AI-assisted code analysis is described only at the outcome level (4 h, per-system tickets); there are no details on the tool, prompts, or the precision/recall of its analysis.
  • eMudhra's root-cause script, and the role of auditors and cross-signers, are still open questions at the time of writing.

Source

Last updated · 854 distilled / 2,448 read