Skip to content

CONCEPT Cited by 1 source

Inner container VM

Definition

A Linux container slid between user code and the VM's kernel, separating:

  • an outer / root namespace where platform-owned orchestration services run, and
  • an inner / user namespace that user code sees and inhabits as its execution environment.

From the user's point of view they have a root shell on a Linux VM. From the platform's point of view they have a container inside the VM, and the platform can start / stop / bounce / snapshot / restore the user's view without rebooting the VM kernel underneath it.

Canonical wiki statement

Fly.io Sprites, 2026-01-14:

"User code running on a Sprite isn't running in the root namespace. We've slid a container between you and the kernel. You see an inner environment, managed by a fleet of services running in the root namespace of the VM. […] The inner container allows us to bounce a Sprite without rebooting the whole VM, even on checkpoint restores."

(Source: [[sources/2026-01-14-flyio-the-design- implementation-of-sprites]])

Properties

1. User-facing root, platform-facing sub-root

User code sees a conventional Linux environment and thinks it has root on a VM. Platform code runs "one level out" — it owns the kernel, the root filesystem, the orchestration daemons, and the network bridge. The platform operates on the user's shell from the outside.

2. Bounce without kernel reboot

Restarting the user's container (to recover from a crash, apply a checkpoint-restore, pick up a new image version) is a namespace restart, not a VM reboot. Kernel boot cost amortises across many user-session bounces.

3. Orthogonal lifecycle from the VM

The VM can stay alive across many inner-container lifetimes. Conversely, the inner container can survive across VM migrations (depending on checkpoint scheme). The two lifecycles are independent.

4. Isolation is Linux-container-grade, not VM-grade

The inner-outer boundary is cgroups + namespaces (+ whatever capability / seccomp filters the platform applies), not KVM. The VM boundary is still the user-facing isolation layer between tenants; the inner-container boundary is the isolation layer between user code and platform orchestration code within the same tenant VM.

Relation to inside-out orchestration

The inner container is the substrate that makes concepts/inside-out-orchestration viable. Without an inner container, platform services in the VM root namespace would share the user's namespace, file system, and process table — disastrous for security, coupling, and bounce semantics. The inner container gives the platform a stable side to operate from.

Wiki-adjacent shapes

The idea of an inner container in a VM is not novel in isolation — the novel piece in Sprites is what runs in the outer / root namespace. Some adjacent-but-distinct shapes:

  • Docker-in-VM sandbox harnesses: user code is inside a Docker container that's inside a VM; the outer-VM OS is a minimal sandbox host. Shape matches; but the outer VM is idle infrastructure, not platform-orchestration code.
  • Kata Containers: container-per-VM model (each container gets its own micro-VM). Shape inverts — Sprites use one VM for one container; Kata creates one VM per container as the isolation story.
  • Firecracker + jailer + microVMs-with-agents (AWS Lambda): the VM has a guest agent, but the guest agent is a narrow RPC channel, not an orchestration-services host.
  • Shared-kernel hosting systems (generic Linux containers) — outer layer is the kernel of a multi-tenant host; inner is a user container. Sprites push the outer layer one step in, so each user gets a dedicated kernel + a dedicated set of platform services.

What "bounce" means in Sprites

The Sprites post implies a specific operational meaning for inner-container bounce:

  • Restart the inner container (kill and re-launch the user's namespace / root processes).
  • Keep the VM kernel, the storage stack, the service manager, the log pipeline, and the ingress proxy running.
  • If bouncing is part of a checkpoint-restore, the service manager replays the user's registered services inside the freshly-restored namespace.

Because checkpoint-restore is [[concepts/fast-checkpoint-via- metadata-shuffle|metadata-shuffle-fast]], an inner-container bounce can sit inside the checkpoint-restore path without dominating its latency.

Caveats

  • Threat model undisclosed — the post doesn't enumerate container-escape risk into the root namespace or the authentication between inner and outer.
  • Kernel-version lock-step — because user code shares a kernel with the outer platform, kernel-security updates still require a VM reboot. Bounce-without-reboot is about the user-facing side, not about kernel patching.
  • Resource accounting — cgroup-level accounting is the tool; the post doesn't discuss how platform-side services' resources are carved out from user-reported VM capacity.

Seen in

  • [[sources/2026-01-14-flyio-the-design-implementation-of- sprites]] — canonical wiki statement.
Last updated · 319 distilled / 1,201 read