bhyve on FreeBSD 15.1 gets first working live migration as WIP RFC

bhyve on FreeBSD 15.1 gets first working live migration as WIP RFC

The author, gyptazy, presents a first working state of live migration for bhyve guests on FreeBSD 15.1. A running amd64 guest can move between two hosts over TCP and keep running; it is paused only for the final stop-and-copy phase. The work builds on bhyve's experimental snapshot support. It adds dirty-page tracking in the vmm kernel module, plus a migration protocol in bhyve and bhyvectl. The goal, the author says, is practical: make a real implementation testable and open to discussion instead of debating live migration in the abstract. The code and test tooling are being published as a WIP RFC. The author links an earlier article on the history and limits of bhyve live migration, and a YouTube demonstration of a VM being migrated.

The author gives two reasons it matters. Live migration enables host maintenance, hardware replacement, load balancing and failure-domain management with far less disruption to workloads. It is also central to Sylve, a project that aims to offer a Proxmox-like virtualisation solution for FreeBSD. And it matters for BoxyBSD, where the author offers free BSD-based VPS instances to individuals: without a credible migration path, routine host work can require a guest outage, which the author calls a current show stopper for that project.

How it works: the source starts with a pre-copy phase. It sends populated guest memory, then repeatedly gets a bitmap of pages changed since the previous scan and sends only those pages. Once the remaining dirty set fits the configured downtime target, stops shrinking, or reaches the configured round limit, bhyve pauses the guest and its devices. It then sends the final dirty pages and the existing snapshot state for the vCPUs, kernel state and emulated devices. The destination validates the basic VM geometry, receives memory and state, restores it and reports that it is ready. Only then does the source send GO, the commit point. Before GO, a failed migration can resume the source guest, and the destination does not start the guest if it never receives the commit message. A warm, stop-and-copy mode also exists, as a fallback and a comparison baseline.

What is implemented today: configurable live pre-copy with downtime and maximum-round controls, plus explicit stop-and-copy migration. The handshake carries the vCPU count and the destination refuses a mismatch; all vCPUs are paused and resumed during migration. Memory consistency is checked with a workload that rewrites and verifies guest memory and flags any missing or stale page, and repeated ping-pong migrations were used to confirm the workload keeps running without errors. Disk writes are checked with a guest workload that repeatedly writes and fsyncs files, verifies checksums and is checked again after a cold reboot. Failure handling was tested for a destination that refuses, an unreachable target, and a target that disappears during pre-copy and during stop-and-copy. The destination checks protocol version, page size, memory layout and vCPU count before data transfer. The migration socket is opened by bhyvectl and passed into bhyve so bhyve can stay inside its Capsicum sandbox. The scope is amd64 only and depends on the experimental BHYVE_SNAPSHOT support. It relies on snapshot-capable emulated devices, and PCI passthrough is not migratable yet.

Observed results: in the author's two-host test environment, repeated live migrations, memory-rewrite workloads and fsync-heavy disk workloads completed successfully. For a slowly dirtying 512 MiB guest, downtime was roughly 138 to 171 ms. A rapidly dirtying guest may not converge during pre-copy, but still falls back to a correct stop-and-copy completion with a longer pause. The published test bed has mainly exercised a one-vCPU guest, so multi-vCPU stress and race testing are still required before broader claims. The author calls these encouraging engineering results, not a production support promise or a complete correctness proof.

The post lists seven blockers to address before upstreaming into FreeBSD can be evaluated, described as neutral engineering requirements. First, prove and harden dirty-log correctness: the kernel dirty-page algorithm must be shown not to lose a write while dirty bits are examined and cleared, with concurrency, paging, emulation-write and multi-vCPU stress coverage and a reviewable locking and memory-ordering argument. Second, define a complete compatibility contract: matching memory size and vCPU count is not enough, and a destination should verify guest identity, CPU capabilities, bhyve/vmm and snapshot format compatibility, device topology and capabilities, and shared-storage identity. Third, harden the transport and protocol: the current TCP transport has no authentication or encryption and uses a first-version native protocol, so a stable one needs explicit wire formats, bounded message handling, timeouts, version negotiation and authentication or a supported external secure transport model. Fourth, clarify the commit and device-side-effect boundary, so device restoration cannot cause externally visible work before the source commits and an uncertain outcome after a lost post-commit acknowledgement is recoverable. Fifth, specify storage and network handoff: shared storage needs defined coherency, durability and fencing assumptions, and networking needs answers on tap/bridge handling, MAC/IP movement and traffic convergence. Sixth, build an in-tree test strategy with automated regression coverage and a documented support matrix across CPU counts, memory sizes, devices, host CPUs and failure cases. Seventh, split the work into reviewable patches: dirty-page tracking, snapshot API refactoring, protocol and receiver work, CLI changes and tests.

Next steps: the author says deeper discussions have already taken place in the Sylve Discord, and that publishing the current state is meant to make them more concrete, bring in wider FreeBSD and bhyve feedback and turn the prototype into a roadmap. The immediate focus is correctness, compatibility validation, transport safety and repeatable tests, not performance features. Compression, zero-page elision, broader device coverage, richer orchestration and performance tuning can follow once the foundation is sound. According to the post, some people already have access to the code, and it is planned to be published at least as a proposal on the author's git (git.gyptazy.com/gyptazy/freebsd-bhyve-live-migration) and on GitHub (github.com/gyptazy/freebsd-bhyve-live-migration). Anyone who operates bhyve, works on FreeBSD virtualisation, or is interested in Sylve or BoxyBSD is invited to test the RFC and give design feedback.

Key facts

  • A running amd64 bhyve guest on FreeBSD 15.1 can move between two hosts over TCP and is paused only for the final stop-and-copy phase.
  • It builds on bhyve's experimental snapshot support, adding dirty-page tracking in the vmm kernel module and a migration protocol in bhyve and bhyvectl.
  • For a slowly dirtying 512 MiB guest, observed downtime was roughly 138 to 171 ms in a two-host test environment; the test bed mainly exercised a one-vCPU guest.
  • The current TCP transport has no authentication or encryption, and PCI passthrough is not migratable yet.
  • The author lists seven blockers before upstreaming into FreeBSD can be evaluated, from dirty-log correctness proofs to splitting the work into reviewable patches.

Why it matters

Live migration is a building block for enterprise virtualisation: host maintenance, hardware replacement, load balancing and failure-domain management with far less disruption to workloads. The author ties it to two FreeBSD-based projects. Sylve aims to offer a Proxmox-like virtualisation solution for FreeBSD, where migration is not an optional convenience but one of the capabilities that makes clustered operations practical. For BoxyBSD, the author's free BSD-based VPS service, the lack of a migration path means routine host work can force a guest outage. This is the first working state of the feature the author presents for bhyve on FreeBSD 15.1, published as a WIP RFC.

Who it affects

People who run bhyve, work on FreeBSD virtualisation, or follow Sylve or BoxyBSD are the audience the author names, and testing the RFC and giving design feedback is what the author asks of them. Today it concerns testers and contributors, not production operators: the post says it is not a production support promise.

How to use it

There is nothing to install from a release. The post says the code and test tooling are being published as a WIP RFC, that some people already have access, and that publication is planned at least as a proposal on the author's git and on GitHub under freebsd-bhyve-live-migration. The work targets FreeBSD 15.1 on amd64 and needs the experimental BHYVE_SNAPSHOT support. The implementation offers configurable live pre-copy with downtime and maximum-round controls, and a warm stop-and-copy mode. A demonstration video is on the author's YouTube channel.

How solid is it

The results come from the author's own two-host test environment. Repeated live migrations, memory-rewrite workloads with verification and fsync-heavy disk workloads with checksums completed successfully, and failure cases (refusing destination, unreachable target, target disappearing mid-transfer) were tested. Downtime was roughly 138 to 171 ms for a slowly dirtying 512 MiB guest. The author calls these encouraging engineering results, not a production support promise or a complete correctness proof. No multi-vCPU test results are reported; the test bed mainly exercised a one-vCPU guest. The hardware, network speed and guest OS of the test environment are not described. No statement from the FreeBSD project or bhyve maintainers is quoted.

Risks and caveats

The author's own blocker list is the best guide. The dirty-page algorithm still has to be shown not to lose a write, and multi-vCPU stress and race testing remain required. Compatibility checks cover only protocol version, page size, memory layout and vCPU count. The TCP transport has no authentication or encryption. Storage and network handoff (shared-storage coherency and fencing, tap/bridge handling, MAC/IP movement) are not yet specified, and an in-tree test strategy and a split into reviewable patches are still needed. Scope is amd64 only, relies on the experimental BHYVE_SNAPSHOT support and on snapshot-capable emulated devices, and PCI passthrough is not migratable. A rapidly dirtying guest may not converge in pre-copy and falls back to stop-and-copy with a longer pause. No release date or timeline is given for upstreaming into FreeBSD or for inclusion in a FreeBSD release.

“These are encouraging engineering results, not a production support promise or a complete correctness proof.”

— gyptazy, author of the post