Rewind VM debugger gets most of its features free from Nix derivations

A few days after writing about Rewind VM, a deterministic VM in which every run of a Nix build is a pure function of its inputs, thread schedule included, the author (GitHub handle fzakaria) describes how the tool's debugger has grown. The author says Rewind has been used to find, reproduce and solve numerous race conditions in Nix builds. New features include a source panel, stack frames, bookmarks, a Compare tab, much more gdb support, thread lanes that show who held the CPU at every step, and "Check from here", which helps find the exact step where a race happens. The recurring discovery, in the author's words, was that the hard part of each feature was already done, and Nix had done it. A debugger needs a program's exact inputs, debug symbols, sources, the sources of every library beneath it, and a way for someone else to get all of that on their machine. That is what a derivation is.

The running example is a classic race: two tellers (threads) depositing into one bank account. Each deposit reads the balance, writes a ledger line, then stores balance plus deposit. If one teller runs between the other's read and store, it overwrites the other's deposit with a stale balance. On the author's 16-core laptop the bank program lost money in 396 of 1,000 runs. Pinned to a single core with taskset, it lost money in none of 1,000, because one core rarely switches threads in the middle of a deposit. Rewind's VM has one CPU, so its first run passes too. That is why Rewind perturbs the schedule: rewind check reruns the build under schedules that ask the guest kernel to reschedule at different steps and narrows the first failure to one step. On the bank example it reported that step 3237 decides it. The passing run and the failing run (schedule 1 over steps 3237..3238) are the same run until that step, and the check took 11 seconds on the author's laptop.

The author then lists what Nix supplies for free. First, the inputs: a derivation's inputs (source, compiler, libraries, kernel, VM configuration) are known to Nix, so rewind nix realises them, packs the closure into a read-only erofs image and boots the VM on it. A run's id is a hash of its inputs, like a store path, and rewind show prints the command that recreates it. When a build succeeds, the guest reports each output's NAR hash and rewind nix checks it against the host's copy and against the binary caches Nix substitutes from, fetching only the .narinfo, which validates that the VM build matches the build on the laptop.

Second, symbols and sources. nixpkgs builds packages with separateDebugInfo and the debug info is cached on cache.nixos.org as a debug output. Using debuginfod, Rewind fetches debug info and sources by build ID, so it can show the source of any binary in the VM, the Linux kernel included. A panel in the Rewind app, or rewind where in the terminal, shows the source at the playhead and the calling stack frames; the example shows the deposit function at bank.c:24 called from teller and start_thread.

Third, gdb on a fork of any step. rewind gdb opens gdb on a fork of the run at the playhead with every thread of the process, supporting breakpoints, watchpoints and inspection of memory, registers and variables. At step 3249, just before teller 2 gets the CPU back, one can watch the balance variable and continue until it changes. Nothing done in gdb changes the recording, so the run can be rewound and forked again at the same or any other step. rewind shell --with nixpkgs#strace opens a shell in the VM at a step with any nixpkgs package on its PATH, as one more closure packed into one more image.

Fourth, comparison and thread lanes. The Compare tab, or rewind compare, shows the last shared events of two runs and the first event that differs; teller 2 starts from a balance of 150 in the failing run and 200 in the passing one. Because the VM has one CPU, exactly one thread runs at any step, but the trace records what threads did, not who was running in the gaps. Rewind fills the gaps without recording anything new: every run replays exactly, so it can walk a stretch one step at a time and ask the guest kernel which thread is on the CPU. In the failing run, teller 1 (Thread 140) is interrupted at step 3238, halfway through a deposit, having read the balance but not stored it. Teller 2 (Thread 141) gets the CPU for the single step 3239, long enough to read the balance, and then teller 1 gets it back and finishes.

Fifth, "Check from here". A single bad interleaving raises the question of how likely it is, and whether the passing run was normal or lucky. rewind check normally answers by rebuilding the whole derivation under many schedules. With --run, it instead starts from an existing run at a step you pick, forks it once per schedule with a different thread order from that step on, and counts how many end differently; everything before the step stays as it was. Starting from step 3,221, where the two runs diverged, 16 schedules were run and all 16 lose money (schedule 0 is the original passing run; the forks exit with code 2 because make check fails when the balance comes up short). The output reads "16 of 16 perturbed schedules ended differently". The same is available in the app from the playhead's context menu. Bookmarks (the b key) attach a note to a step, are kept with the run and travel in its .rwd export, so whoever receives the run sees the notes on the timeline.

The flake includes more examples, each a derivation with one bug and one fix: philosophers (the deadlock from the previous post), bank (the lost update), waiter (a SIGCHLD that arrives between a flag check and pause, so the parent sleeps until make check's ten second timeout kills it), and config-reload (one process rewrites a config file in place while another rereads it; on many cores it fails nearly every time, on one CPU only some schedules land the reader between the truncate and the last write). The author's conclusion: most of a debugger's hard work is already done by Nix, and when you start from something hermetic like a derivation, you can get a debugger for free.

Key facts

Rewind VM is a deterministic VM where every run of a Nix build is a pure function of its inputs, thread schedule included; new debugger features include a source panel, stack frames, gdb on a fork of any step, a Compare tab, thread lanes, "Check from here" and bookmarks.
The author's claim: the hard part of each feature was already done by Nix, since a derivation already carries exact inputs, debug symbols and sources, and nixpkgs debug info is cached on cache.nixos.org and fetched through debuginfod by build ID.
On the bank-account race, the author's 16-core laptop lost money in 396 of 1,000 runs, but pinned to one core with taskset in none of 1,000; Rewind's single-CPU VM passes the first run, so rewind check perturbs schedules and found the deciding step 3237 in 11 seconds.
"Check from here" forks an existing run at step 3,221 under 16 schedules and all 16 lose money, answering how likely the bug is without rebuilding the whole derivation.
Nothing done in gdb changes the recording, so a run can be rewound and forked again at any step, and runs can be exported as .rwd files with bookmark notes.

Why it matters

Race conditions are hard to debug because they are hard to reproduce: in the author's example the bug showed up in 396 of 1,000 runs on a multi-core laptop and never on a single core. Rewind's pitch is that a Nix build becomes a fully replayable run, so a flaky failure can be pinned to one step and examined at leisure. The post's broader argument is about reuse: because a Nix derivation already carries exact inputs, debug symbols and sources, a debugger built on top of it needs far less new machinery. The author says each new feature turned out to be less of a code-lift than expected.

Who it affects

Mainly people who build software with Nix and fight intermittent failures, such as races and deadlocks, in those builds. The author says Rewind has been used to find, reproduce and solve numerous race conditions in Nix builds. It is also a data point for anyone designing debugging or reproducibility tooling on top of hermetic build systems.

How to use it

The post gives the entry point: nix run github:fzakaria/rewindvm -- check github:fzakaria/rewindvm#bank. The flake carries four example derivations, each with one bug and one fix: philosophers, bank, waiter and config-reload. Terminal commands shown include rewind check (with --where to narrow to a step, or --run with --schedule-from and --schedules to fork an existing run), rewind show, rewind where, rewind gdb, rewind shell --with nixpkgs#strace and rewind compare. The Rewind app offers the same through a source panel, a Compare tab, thread lanes and a "Check from here" context menu entry; the b key sets a bookmark.

How solid is it

This is a first-person post by the tool's author, backed by command output on small demo programs rather than an independent evaluation. The specific figures (396 of 1,000, none of 1,000, 11 seconds, 16 of 16) come from one 16-core laptop of the author, and no other hardware or wider benchmark is reported. The step numbers, run ids and schedule results are shown as terminal output, so the bank-account walkthrough is concrete and traceable. The claim that Rewind has solved numerous real race conditions in Nix builds is stated without a list of which ones.

Risks and caveats

No performance overhead, slowdown or resource cost of running builds inside Rewind VM is given. No release date, version number, license or adoption by other users is stated, and the post does not say whether Rewind works on builds that are not Nix derivations. The demos are small, deliberately buggy programs; the 16 of 16 result is for one chosen starting step in one example. Rewind's single-CPU VM passes the bank example on its first run, so finding a race depends on the perturbed schedules rather than on the first run alone.

“When you start with something hermetic like a Nix derivation, you can get a debugger for free.”

— fzakaria, "Nix wrote half of my debugger"