Skip to content

FLYIO 2024-03-07 Tier 3

Read original ↗

Fly.io — Fly Kubernetes does more now (FKS beta)

Summary

Fly.io announces the beta of Fly Kubernetes (FKS) — their "blessed path" managed Kubernetes service. Architecturally, FKS is not a conventional managed-K8s offering (no Nodes, no CRI, no CNI plugin graph). Instead, it pairs K3s as the K8s API/control plane with Virtual Kubelet as a stand-in for the Kubelet/Node; a small in-house Go Virtual-Kubelet provider translates Pod-creation requests into Fly Machines (Firecracker micro-VMs) via the Fly Machines API. Every Kubernetes primitive is mapped 1:1 (or close) to an existing Fly.io primitive: containerd/CRI → flyd + Firecracker + Fly init; CNI → the internal IPv6 WireGuard mesh; Pods → Fly Machines; Services → the Fly Proxy; Secrets → Fly Secrets; Persistent Volumes → Fly Volumes (coming soon); CoreDNS stays CoreDNS for now. The deliberate architectural choice is that "there's no Node" — the cluster operator never picks node shapes or runs capacity planning. Pricing signal: $75/mo per cluster post-beta, plus usage. FKS supports Deployments, ReplicaSets, Services (with standard K8s DNS), and ephemeral + persistent volumes; missing at beta: multi-container pods, StatefulSets, NetworkPolicies, HPA, emptyDir. Fly explicitly concedes the conformance question: "this isn't Kubernetes!" — we agree!" — and flags future conformance tests as maybe-later.

Key takeaways

  1. Nodeless Kubernetes via Virtual Kubelet is the core architectural trick. Virtual Kubelet "acts as if it's a standard Kubelet running on a Node" but has no Node behind it — it receives Pod-creation requests from Kubernetes and forwards them to a cloud compute service. Fly runs a small in-house Go Virtual-Kubelet provider alongside K3s that calls Fly Machines API to spin up pods as micro-VMs. Net effect: "this shifts the burden of managing hardware capacity from you to us." (Source: this article)

  2. One-to-one primitive mapping from Kubernetes to Fly.io. The post lists the current mapping explicitly (patterns/primitive-mapping-k8s-to-cloud):

  3. Containerd / CRI → flyd + Firecracker + the Fly init ("our system transmogrifies Docker containers into Firecracker microVMs")
  4. Networking / CNI → internal IPv6 WireGuard mesh connecting pods together (systems/fly-wireguard-mesh)
  5. Pods → Fly Machines VMs
  6. Secrets → Fly Secrets ("only not the base64'd kind")
  7. Services → the Fly Proxy
  8. CoreDNS → CoreDNS ("to be replaced with our custom internal DNS")

  9. Persistent Volumes → Fly Volumes (coming soon) Explicitly not a bidirectional mapping: NetworkPolicy, init containers, multi-container pods, StatefulSets, HPA, emptyDir are "not completely ignored" but not supported at beta.

  10. The Pod is a Fly Machine. kubectl run --image=... creates a Kubernetes Pod, which the Virtual Kubelet provider transforms into a Firecracker micro-VM via the Machines API and places on an underlying host in the selected region. Observable via fly machine list --app <fly-app>; each cluster corresponds to a single Fly App labeled fly.io/app=fks-default-<cluster-id> on the default Namespace. This is a concrete instance of concepts/micro-vm-as-pod — Firecracker is the isolation substrate under what the K8s API calls a Pod, identical-in-spirit to the AWS Fargate-on-Firecracker posture but with Fly's own orchestrator (systems/flyd) and scheduling surface.

  11. Services work, but the underlay is IPv6 + flycast, not iptables + ClusterIP-over-IPv4. kubectl expose pod ... --port=8080 creates a standard ClusterIP Service; the Service resolves to an fdaa:… IPv6 address (IP Families: IPv6, IP Family Policy: SingleStack). Three ways to reach it:

  12. Direct IPv6 from inside the org's 6PN private WireGuard network: http://[fdaa:0:48c8:0:1::1a]:8080.
  13. flycast: http://<service_name>.svc.<app_name>.flycast:8080 (internal private load balancing).

  14. CoreDNS inside the cluster: <service_name>.<namespace>.svc.cluster.local resolves to the fdaa:… IP and is routable within the cluster. A concrete annotation on the Service — fly.io/clusterip-allocator: configured, service.fly.io/sync-version: 11507529969321451315 — exposes that FKS has a bespoke allocator / sync loop wiring K8s Service objects into the Fly Proxy (systems/fly-proxy).

  15. Explicit leaky abstraction at the kubectl exec / port-forward boundary. "We don't yet support all kubectl features… we don't have kubectl port-forward and kubectl exec, but you can use flyctl to forward ports and get a shell into a pod." That is, some classical Kubelet responsibilities (proxying a stdio channel, opening a port-forwarded socket) simply aren't implemented in the Virtual-Kubelet provider yet; users drop to the Fly-native tool (flyctl) because the Pod is a Fly Machine anyway. This is the honest face of a Virtual-Kubelet-based K8s: some parts of the API surface are not free once you remove the Node.

  16. GPUs and multi-region are first-class. "You can deploy your workloads (including GPUs) across any of our available regions using the Kubernetes API." The region is selected at cluster-creation time (fly ext k8s create --name hello --org personal --region iad); scheduling inside a region is "to any underlying host" — so Fly owns the bin-packing (concepts/empty-host-adjacent behaviour is Fly's problem, not the cluster operator's).

  17. Beta API-surface gaps are named explicitly. Unsupported at beta: multi-container pods, StatefulSets, NetworkPolicies, HPA, emptyDir volumes. Roadmap: HPA + emptyDir "in the coming weeks", multi-container pods "in the coming months". These gaps matter because a large fraction of real-world K8s charts (sidecars, init-containers, any StatefulSet workload, anything relying on network-policy isolation) won't run unmodified.

  18. The conformance question is acknowledged head-on. "If you've made it this far and are eagerly awaiting your chance to tell us and the rest of the internet 'this isn't Kubernetes!', well, we agree! It's not something we take lightly. We're still building, and conformance tests may be in the future for FKS. We've made a deliberate decision to only care about fast launching VMs as the one and only way to run workloads on our cloud." The post frames FKS as "Kubernetes API → fast-launching VM" rather than conformance-first. Companion framing: "a fast launching VM in the form of a Pod" — the Pod is the compatibility surface, the VM is the substrate.

  19. Pricing signal and operational footprint. Free during beta; $75/mo per cluster after, plus the usual usage cost of Fly Machines and Fly Volumes ("same as for your other Fly.io projects"). No disclosure of per-Pod scheduling latency, per-region capacity, cluster size limits, or K3s control-plane sizing.

Architecture at a glance

                      Kubernetes API (K3s)
                             │
                             ▼
                     Virtual Kubelet
                (Go program, alongside K3s)
                             │
           "fake Node" eager to run pods │ provider API
                             ▼
                      Fly Machines API
                             │
                             ▼
             flyd + Firecracker + Fly init
                             │
                 ┌───────────┴───────────┐
                 ▼                       ▼
          Pod = micro-VM          IPv6 WireGuard mesh
                                   (CNI replacement)
                                         │
                             Services → Fly Proxy
                               (+ flycast, CoreDNS)

Operational numbers / disclosures

  • $75 / month / cluster post-beta pricing.
  • Cluster creation command: fly ext k8s create --name <name> --org <org> --region <region>.
  • Each cluster = one Fly App (fly.io/app=fks-default-<cluster-id>), with the default namespace pre-created.
  • Service ClusterIPs are IPv6 only (fdaa:…).
  • No disclosure of: Pod-create latency, cluster-scale ceilings, K3s control-plane HA posture, cross-region cluster support, GPU-VM scheduling latency, capacity-reservation semantics.

Caveats

  • Fly.io is a Tier-3 source on this wiki. The post is borderline launch-announcement but carries load-bearing architectural content: the Virtual-Kubelet + Firecracker + WireGuard-mesh composition is a concrete instance of patterns/virtual-kubelet-provider and concepts/nodeless-kubernetes worth capturing.
  • Beta posture. Multi-container Pods, StatefulSets, NetworkPolicy, HPA, emptyDir volumes are not supported at the time of writing; roadmap commitments are in weeks / months.
  • CoreDNS is retained at beta but flagged for replacement ("to be replaced with our custom internal DNS") — the service-discovery substrate may shift under FKS users mid-beta.
  • No conformance claim. Post explicitly agrees with skeptics that "this isn't Kubernetes!" in the CNCF-conformance sense.
  • No published numbers for Pod-create latency, cluster-size ceilings, or failure-mode behaviour. Beta-cohort production experience not surfaced.
  • The fly machine list example output shows shared-cpu-1x:256MB as the resulting Machine size; the mapping from K8s resources: requests/limits to Fly Machine shape is not disclosed here.

Source

Last updated · 178 distilled / 1,178 read