Skip to content
pilots
Dashboard GitHub (opens in a new tab)

No control plane, on purpose

Every host runs the identical stack and serves the entire API. There is no scheduler to register with, no database to fail over, and no appliance in front. The tradeoffs that choice forces are the interesting part, and they are all below.

The internals, with diagrams Read ARCHITECTURE.md (opens in a new tab)

Five things that are never traded away

Several of these were paid for with production incidents in the codebase that came before this one. They are written down because the expensive ones are the ones that look optional right up until the moment they are not.

  1. 1

    The data plane never depends on the control plane

    Routing and wake read local state only. There is no scheduler tier, no managed database, and no load balancer appliance. No request path may require a specific machine to be alive, and a change that breaks that is not a regression to be tuned, it is the wrong change.

  2. 2

    A host writes only its own rows

    State is a CRDT with last-write-wins merges, which means there are no uniqueness constraints and no cross-host transactions to lean on. Two hosts writing the same row does not error. It corrupts silently, which is why this one is enforced in review rather than by the database.

  3. 3

    Object storage is the only truth

    Local NVMe is a cache and nothing more, for a running machine’s root disk and for a volume alike. There is one storage model rather than a local disk plus a network one. The design test is blunt: wipe any host’s disk and nothing is lost. Anything that fails that test is state living in the wrong place.

  4. 4

    URLs are permanent

    An address survives suspend, wake, checkpoint, restore, promote, redeploy, and migration to another host. Any change that can mint a new URL for an existing machine is a bug, not a tradeoff.

  5. 5

    Snapshots are host-agnostic

    Nothing host-specific may enter a snapshot. That single requirement is why guest networking uses constant addresses and why the rootfs is bind-mounted to a constant path, both described below.

Three processes, and that is the whole machine

A pilots host is not a node in a cluster that something else manages. It runs 3 processes, holds a full replica of the fleet’s state, and can answer any API call for any machine in the fleet, including ones it has never run.

hostd

The entire data plane in one Go binary: the API, the router and its TLS, the Firecracker supervisor, the block layer, the idle monitor, and the self-heal loop.

corrosion

Gossip-replicated SQLite. Every host reads its own local replica, so a lookup on the request path is a local read rather than a network call to something that might be down.

firecracker

One process per machine, inside its own network namespace, under the jailer with a cgroup slice bounding CPU, memory, and process count.

Why a CRDT forces the single-writer rule

Gossiped state buys availability: no host waits on a quorum, and a partitioned host keeps serving. What it costs is arbitration. Last-write-wins means two hosts editing one row produce a merge rather than a conflict, and the loser vanishes without an error anywhere. So ownership is assigned instead of contested, and anything genuinely needing a single actor is resolved by hashing rather than by electing:

deterministic ownership 
owner(key) = hash(key) mod live_hostsname allocation, self-heal slices, build assignment

Every host computes the same answer from the same inputs, so they agree without talking. That is the whole coordination mechanism, and it is why there is no leader to lose.

A snapshot that does not know which host made it

A memory snapshot is only portable if nothing host-specific got baked into it, and the two places that happens are networking and file paths. Both are solved by making the guest’s view of the world constant, and keeping the parts that genuinely differ outside the snapshot.

Constant addresses

Every guest, on every host, sees the same network: the same address on the same interface behind the same gateway. The per-slot addressing that actually routes packets exists only in the namespace’s translation rules, which are rebuilt at restore and never enter the snapshot. Each host carries 1024 such slots.

A constant path

A snapshot bakes in the absolute path of its disk, and sharing one rootfs file between machines causes lockups after resume. So each machine gets a block device of its own, mounted onto the same path inside its own mount namespace. No machine holds a copy of the disk behind it. The path is the invariant, not what sits there, and every snapshot restores against a path that exists identically everywhere.

Content-addressed, so most of it is never stored

Memory and disk are both chunked into 4KiB blocks. A block that is all zeroes is recorded as a gap and occupies nothing. A block identical to the template it came from is recorded as a pointer at the template and occupies nothing. Only genuinely divergent blocks are stored, which is why a checkpoint of a machine that changed little uploads little, and why a machine that changed nothing skips the upload entirely.

Chains are exactly two levels deep, a template and one diff. A reference to a grandparent is rejected when the header is parsed rather than discovered later as a page that resolves to the wrong bytes.

One storage model, and the host disk is not in it

Platforms that pin a disk to a host spend years making that disk movable. pilots never pins one. The machine root and the volume are both true only in the bucket, and the host’s NVMe is a read-through cache in front of it that can be emptied at any time.

The machine root

The truth of a root disk is its chunked build chain in object storage. A host serves it as a block device over a template every machine on that host shares, with the machine’s own writes kept as dirty blocks. A host with a cold cache serves the root from the bucket while it fills, rather than waiting for a copy. Durable as of the last checkpoint, suspend, or flush, and a flush runs at most 60s apart.

The volume

A filesystem whose blocks live in the same bucket, with its index replicated there continuously. Durable per write, with write-back buffering deliberately off, which is why a database belongs here and not on the root. It follows its machine to whichever host runs it next, because it was never local to the last one.

The two promises are stated separately on purpose. A root that promised per-write durability would pay an upload on every write to a scratch file, and a volume that promised a window would be the wrong place for a ledger. A root flush pauses the guest for the blocks written since the last one, budgeted at 25ms, which is small and is not zero. That is the one place this design trails a copy-on-write flush that never stops the guest, and it is measured rather than hidden. The internals page draws the three places a byte can be.

A request to a sleeping machine waits while it wakes

The router lives inside the same binary that supervises the microVMs, which is what makes waking on demand a local operation rather than a distributed one. A request arrives for a machine that is suspended, and the connection is simply held open while it comes back.

  1. 1

    Terminate TLS

    A wildcard certificate for platform addresses, per-domain certificates issued on demand for custom ones. Any host can answer the challenge, so issuance is not pinned anywhere.

  2. 2

    Resolve the name

    Parsed from the hostname, then looked up in the local replica. Microseconds, no network.

  3. 3

    Route or wake

    Running here: proxy straight into the namespace. Running elsewhere: proxy over the mesh to the host that owns it. Suspended: hold the connection, restore, then proxy.

  4. 4

    Record activity

    Every request and every exec touches the machine’s last-activity stamp, which is what the idle monitor reads.

Suspension needs both of its conditions to agree. The idle timer (60s by default) has to expire and there has to be nothing in flight. An agent partway through a long build generates no HTTP traffic at all, and a timer alone would put its machine to sleep underneath it.

A dead host is noticed by everyone at once

Every host writes a heartbeat. After 30s of silence a host is presumed dead, and every survivor independently rescues the slice of its machines that hashes to its own index. There is no election because there is nothing to elect.

fsn1-a
  • bold-otter
  • quiet-finch
fsn1-b
  • lucky-moth
  • plain-heron
nbg1-a
  • brisk-vole
  • north-elk

All hosts healthy. Kill one and its machines return on the survivors, same URLs.

The rescued machines rebuild from object storage, which is the reason this works at all: nothing needed from the dead host, because nothing authoritative was ever only there. That now covers the machines that were running as well as the ones asleep. A sleeping machine wakes from its snapshot, and a running one cold-boots from its last flushed disk with the same id, name and address, having lost at most the flush window. The slices tile the dead host’s machines with no overlap and no gaps, so the survivors do not need to agree with each other, only to run the same function.

Placement can still be refused. A host is the final authority on its own capacity, so a rescue aimed at a full host is rejected and re-hashed rather than accepted and then failed. Coordinators propose. Hosts dispose.

What this design costs

Choosing no control plane is not free. These are the bills it comes with, and they are structural rather than temporary.

CPU vendor pools, not one fleet

Memory snapshots carry raw CPUID and will not cross the Intel/AMD line, so a snapshot is portable within a vendor pool and never across one. A machine whose pool has no live host does not fail: it cold-boots from its own disk, keeping its id, its name, its URL and every byte it had written.

No cross-host transactions

The state layer cannot express one. Anything needing uniqueness has to be reachable by a hash instead, and anything that cannot be is a design problem rather than a query problem.

Silent corruption is the failure mode

A single-writer violation produces no error at all. It merges. Review is therefore where this gets caught.

Every host is a security boundary

Since every host serves the full API, every host authenticates. Key hashes are replicated so authentication survives losing any host, including the one running the dashboard.

If this is the kind of thing you want to argue with, the design is written down in full and the code is next to it. The internals page draws all of it, one mechanism at a time, and the roadmap says which parts are finished.