SYSTEM Cited by 4 sources

flyd

flyd is Fly.io's in-house scheduler / orchestrator for Fly Machines (Firecracker micro-VMs). Carved out of Fly's earlier Nomad-based orchestrator per Carving the scheduler out of our orchestrator. Under FKS it stands in for the usual CRI layer that a K8s worker node would expose: when the Virtual-Kubelet provider asks to create a Pod, flyd + Firecracker + the Fly init together do the work that containerd + runc would do on a standard Node.

Role in FKS

Per the primitive-mapping table in the FKS beta announcement:

Containerd / CRI → flyd + Firecracker + our init

flyd receives a Machine-create request via the Machines API (invoked by the Virtual-Kubelet provider), places the Machine on a host in the target region, boots it via Firecracker, and wires it into the IPv6 WireGuard mesh. The Virtual Kubelet never sees containerd, runc, or a CNI plugin chain.

Seen in

• sources/2024-03-07-flyio-fly-kubernetes-does-more-now — named as the CRI substitute under FKS ("flyd + Firecracker + our init"); the component that turns Docker images into Firecracker micro-VMs on Fly's hosts.
• sources/2024-07-30-flyio-making-machines-move — flyd's durable-FSM model is the load-bearing architectural substrate for stateful Machine migration. "Conceptually, flyd is a server for on-demand instances of durable finite state machines, each representing some operation on a Fly Machine (creation, start, stop, &c), with the transition steps recorded carefully in a BoltDB database. An FSM step might be something like 'assign a local IPv6 address to this interface', or 'check out a block device with the contents of this container', and it's straightforward to add and manage new ones." The migration FSM added for 2024 stateful-drain is one more such state machine — with a new step carrying LUKS2 header metadata from source to target worker (patterns/fsm-rpc-for-config-metadata-transfer). The post also positions this work relative to flyd's own origin: "This is the biggest thing our team has done since we replaced Nomad with flyd."
• sources/2025-02-12-flyio-the-exit-interview-jp-phillips — exit interview with the engineer who led flyd, four years in. Three new wiki-quality disclosures:
• FSM design ancestry: "I think the FSM stuff is a result of work I did at Compose.io / MongoHQ (where it was called 'recipes' / 'operations') and the work I did at HashiCorp using Cadence." The durable-execution idea is not novel to flyd; the specific BoltDB embodiment is. Cadence is lineage, not a runtime dependency.
• Deploy-tolerance is the motivating constraint: "One of the biggest gains, with how it works in flyd, is knowing we would need to deploy flyd all day, every day. If flyd was in the middle of doing some work, it needed to pick back up right where it left off, post-deploy." Canonical wiki citation for why flyd needs durable execution, not just an FSM library.
• BoltDB-over-SQLite was deliberate: JP stands by the choice three years in. "I've never lost a second of sleep worried that someone is about to run a SQL update statement on a host, or across the whole fleet, and then mangled all our state data. And limiting the storage interface, by not using SQL, kept flyd's scope managed." See concepts/bolt-vs-sqlite-storage-choice for the full decomposition.

The same interview discloses pilot — Fly's next-generation init — as the successor to the feature-bag init, with a defined API for flyd to drive it. This restores a formal contract at the init/orchestrator boundary that Fly had let drift for years. The related flaps Machines-API gateway is named as part of the "whole Fly Machines API", with a disclosed sub-5-second P90 on machine create globally (Johannesburg and Hong Kong excepted) — architecturally notable because flaps routes to per-host flyd RPCs "for the most part [without] any central coordination."

• sources/2025-02-14-flyio-we-were-wrong-about-gpus — flyd orchestrates GPU Machines on a different hypervisor. Non-GPU Fly Machines run on Firecracker; GPU Fly Machines run on Intel Cloud Hypervisor (PCI passthrough support). flyd's scheduler carries the workload-class split — a Machine requesting a PCI BDF for a GPU lands on a GPU-dedicated worker on Cloud Hypervisor; other Machines land on Firecracker workers. Canonical instance of patterns/dedicated-host-pool-for-hostile-peripheral at the scheduler-placement tier. Also: the 2025-02-14 post names the engineering cost of GPU Machines inside Fly's stack in passing: "we had to engineer around [the Fly-Machine root-filesystem assumption] in our flyd orchestrator" to get Nvidia drivers into the guest.

JP's alternate design (2025-02-12): per-Machine SQLite

Not built — but recorded for the wiki as the alternative JP would consider if re-doing flyd today:

I'd maybe consider a SQLite database per-Fly-Machine. Then the scope of danger is about as small as it could possibly be. … With per-Machine SQLite, once a Fly Machine is destroyed, we can just zip up the database and stash it in object storage. The biggest hold-up I have about it is how we'd manage the schemas. (Source: sources/2025-02-12-flyio-the-exit-interview-jp-phillips)

See patterns/per-instance-embedded-database for the general pattern. flyd today is per-host BoltDB, not per-Machine SQLite.

Open questions (not disclosed in the FKS post)

• Scheduler architecture details (bin-packing algorithm, failure recovery, reconciliation cadence).
• Host-level isolation and multi-tenant packing policies.
• How quota / capacity limits are enforced per org / per cluster.