Skip to content

FLYIO 2025-10-08 Tier 3

Read original ↗

Fly.io — Kurt Got Got

Summary

Fly.io's 2025-10-08 post-incident write-up on an account-takeover (ATO) of its @flydotio X/Twitter account via a phishing attack targeting CEO Kurt Mackey. The attack worked because the Twitter account sat outside Fly.io's "things we take seriously" boundary — credentials lived in 1Password as a shared legacy account, not behind Fly.io's Google-backed SSO + phishing- resistant MFA that protects the rest of the production infrastructure. The phish convincingly impersonated a Twitter content-moderation alert about a recent dank-meme post by an intern, triggering Kurt to pull the 1Password credential and log in to a lookalike origin (members-x.com driven from alerts-x.com). The attacker revoked tokens and reset 2FA immediately; Fly.io detected the ATO "45 seconds" after it happened but X/Twitter's account-recovery path took ~15 hours to complete a 2FA reset and lock the attacker out. Blast radius was contained — a low-yield crypto scam, attackers deleted Fly's tweet history ("don't threaten us with a good time"), no customer impact. Core architectural lesson distilled: phishing-resistant authentication (FIDO2/WebAuthn/Passkeys) behind an SSO IdP is the defence against phishing, not training users not to click; hygiene backstops (the 1Password browser plugin would have refused to autofill members-x.com for an x.com entry) are secondary. Remediations: the Twitter account now uses Passkeys; Fly.io will get a real "incident response" population sample for its next SOC2; Kurt loses commit access ("the time comes in the life of every CEO").

Key takeaways

  1. Phishing training is not the defence; phishing-resistant auth is. "You don't defeat phishing by training people not to click on things. I mean, tell them not to, sure! But eventually, under continued pressure, everybody clicks. There's science on this. The cool kids haven't done phishing simulation training in years." The mechanism is mutual / origin- and channel-binding authentication (U2F, FIDO2, Passkeys): the browser refuses to sign a challenge for an origin the credential isn't bound to, so a malicious-proxy phishing site structurally cannot harvest a usable credential. Canonical wiki statement.

  2. Production infrastructure sat behind SSO + phishing-resistant MFA; Twitter didn't. "This is, in fact, how all of our infrastructure is secured at Fly.io; specifically, we get everything behind an IdP (in our case: Google's) and have it require phishing-proof MFA. You're unlikely to phish your way to viewing logs here, or to refunding a customer bill at Stripe, or to viewing infra metrics, because all these things require an SSO login through Google." Canonical instance of patterns/phishing-resistant-mfa-behind-idp — the deployment topology combining IdP federation with a phishing-resistant credential enforced at the IdP.

  3. Legacy shared accounts are the blind spot. "Twitter had been a sort of legacy shared account for us, with credentials managed in 1Password and shared with our zoomer contractor." Account was outside the IdP because Fly.io considered decamping from Twitter in 2023-2024 (many employees moved to Mastodon / Bluesky) — so it never got wired to SSO. Canonical concepts/shared-legacy-account-risk — any account whose credentials are managed outside the SSO+MFA envelope is a latent phishing target regardless of how mature the rest of the auth story is.

  4. The x.com phishing surface is inherently weak. "Kurt complains that 'x.com' is an extremely phishable domain." A one-character TLD makes lookalike domains (alerts-x.com, members-x.com) visually plausible in email-client UIs and browser address bars. Mitigation: password-manager origin matching — 1Password's browser plugin checks the current origin against the stored entry and refuses to autofill a mismatch. "The 1Password browser plugin would have noticed that 'members-x.com' wasn't an 'x.com' host." Weaker than WebAuthn origin-binding (the plugin's check is client-side hygiene, not cryptographic), but still would have caught this attack had it been engaged.

  5. Detection was fast; containment wasn't. Fly.io saw the ATO "45 seconds" after it happened (email notification that the @flydotio account's email address had changed to achilles19969@gmail.com). Password reset was immediate; 2FA reset via X/Twitter's account-recovery path took ~15 hours. The attacker had revoked all tokens and set up their own 2FA in-between, so Fly.io couldn't lock them out without platform intervention. "That's not a knock on X.com; 15 hours for a 2FA reset isn't outside industry norms." Canonical concepts/account-takeover-response — once an attacker replaces 2FA on a platform you don't control, your containment time is bounded by the platform's recovery SLA, not your own.

  6. Immediate response posture: audit credential access, cut, assume endpoint compromise. "Our immediate response was to audit all accesses to the login information in 1Password, to cut all access for anybody who'd recently pulled it; your worst-case assumption in a situation like this is that someone's endpoint has been owned up." Generalises to any shared-credential ATO: audit recent credential fetches → cut access → treat every endpoint that pulled the credential as potentially compromised until cleared. Codified as part of concepts/account-takeover-response.

  7. Blast-radius discipline mattered more than prevention. "Our users weren't under attack, and the account wasn't being used to further intercept customer accounts. At one point, the attackers apparently deleted our whole Twitter history, which, like, don't threaten us with a good time. So we let it roll, until we got our account recovered the next morning." The containment was structural: the @flydotio Twitter account had no path to customer accounts, infra metrics, billing, or any production surface because all of those are behind SSO. Canonical concepts/shared-legacy-account-risk inverse — when the scope of a compromised credential is pre-contained by architecture, the incident is embarrassing but bounded.

  8. Structural remediation: move the outlier inside the envelope. "Either way: our Twitter access is Passkeys now." The one-sentence fix to an ATO caused by an outlier credential is to bring that credential into the phishing-resistant envelope that protects everything else. If the platform supports Passkeys (X/Twitter does), adopt them; if it supports SSO, federate. Canonical patterns/post-incident-retire-shared-credential.

Extracted systems / concepts / patterns

Systems referenced:

  • systems/oidc-fly-io — Fly.io's internal IdP is Google. Extended here with the posture claim that "everything behind an IdP" + "phishing-proof MFA" is the architectural boundary protecting infra, billing, customer data.
  • systems/fly-flyctl — not directly implicated in this incident but is the primary internal-auth-gated surface (logs, Stripe refunds, infra metrics) that the post contrasts with the Twitter outlier.

Concepts (new and extended):

  • concepts/phishing-resistant-mfa — new. WebAuthn / FIDO2 / Passkeys; authentication via mutual / origin- / channel-binding; the browser refuses to sign a challenge for a mismatched origin. Canonical wiki instance.
  • concepts/account-takeover-response — new. The detect → cut-credential-access → assume-endpoint-compromise → wait-on- platform-recovery-SLA sequence.
  • concepts/origin-bound-credential — new. The credential property that's orthogonal to phishing-resistance: the credential is bound to a specific DNS/TLS origin, so cross-origin proxying simply cannot produce a signed assertion for the real origin.
  • concepts/passkey-authentication — extended. Previously canonicalised via the 2026-04-01 Cloudflare EmDash post (passkey-by- default CMS authentication); this source adds the Fly.io deployment angle (Passkeys as the post-incident remediation for a legacy shared social account).
  • concepts/sso-behind-idp — new. The topology of pointing every internal and third-party SaaS through an IdP (Google in Fly.io's case) so that authentication becomes a single federated path a single phishing-resistant credential can defend.
  • concepts/shared-legacy-account-risk — new. Credentials for accounts that predate the SSO policy and live in a password manager shared with third parties (here: a contractor handling Twitter) are latent phishing targets regardless of how mature the rest of the auth posture is.

Patterns (new):

Operational numbers

  • Time to detection: 45 seconds from ATO-initiated until Fly.io employees received the email notification that the account's email had changed to achilles19969@gmail.com.
  • Time to password reset: "quickly" (attacker had already revoked tokens + set new 2FA, so this was insufficient for lockout).
  • Time to platform-assisted 2FA reset: ~15 hours end-to-end via X/Twitter's account-recovery path. Fly.io explicitly flags this as in-line with industry norms.
  • Blast radius (quantitative): $0 attributed to attackers (scam site generated no crypto revenue); ~15 hours of "brand damage"; Fly's Twitter post history was deleted by attackers at one point.
  • Blast radius (scope): 0 customer accounts affected — the Twitter account had no path into customer infrastructure or billing because all those surfaces are behind SSO + phishing-resistant MFA.

Caveats

  • Opinion / retrospective voice, not deep technical teardown. The post is Ptacek's signature snark-plus-substance blend; the architectural substance is high-level ("get everything behind an IdP + phishing-proof MFA") without enumerating specifically which IdP policies, WebAuthn attestation requirements, or account-recovery flows Fly.io's Google deployment uses. Concrete at the framing level, not the configuration level.
  • No enumeration of other still-outlier accounts. The post confesses the Twitter case but doesn't tell us how many other shared legacy accounts in 1Password might still be outside the SSO envelope. Implicit: there is a cleanup project.
  • No Windows-of-exposure detail on the 1Password shared entry. Post says Fly.io audited who had recently pulled the credential but doesn't quantify the sharing cardinality (how many employees + contractors), rotation cadence, or whether the 1Password entry had been accessed from an untrusted device prior to the phish.
  • Attribution is implicit and uncertain. Attacker is not attributed beyond the crypto-scam payload; no IOC sharing, no claims about campaign infrastructure beyond the members-x.com + alerts-x.com lookalike pair and the achilles19969@gmail.com recovery address.
  • Platform-recovery-SLA framing generalises but the specific number is X/Twitter-specific. "15 hours isn't outside industry norms" is asserted, not cross-referenced to other platforms. Users should treat their own platforms' recovery SLAs as a separate risk variable.
  • The 1Password-origin-matching backstop works only if the plugin is actively engaged. If the user types the URL into the address bar and manually copy-pastes the password (not using autofill), the browser plugin never evaluates origin. This is a hygiene feature, not a cryptographic defence, and that distinction matters.
  • Inside a SOC2 framing. Post is partly in the shadow of Fly.io's SOC2 process (ref: SOC2 post) — the "population sample for incident response" joke points at the compliance-test dimension. The architectural posture is consistent with SOC2 requirements but the post is not a SOC2 auditor's writeup.

Source

Last updated · 319 distilled / 1,201 read