Glonce

Google Project Zero releases MAccConc to test Linux kernel race conditions

securityThe Glonce editorsHacker News🔥49Glonce rating49Impact65Novelty70Relevance55Surprise45Players60Credibility88

Race conditions, bugs that only surface under one specific interleaving of multi-threaded execution, are hard to test for in three separate ways: confirming a bug candidate found by manual review or static analysis, writing a regression test that reliably reproduces a fix, and getting a fuzzer to explore the interleavings that trigger the bug at all. The author of a new post on Google Project Zero's blog says they mostly find these bugs by reading code by hand, then try to force the suspected interleaving in the Linux kernel by recompiling it with conditional calls to mdelay(), a busy-wait for roughly a set duration, inserted at the suspect site and usually made conditional on the name of the running thread. On macOS and Windows, DTrace's chill() probes offer something similar, but only at non-inline function boundaries or explicit trace points rather than at every instruction, which limits how finely execution can be paused. The author also points out that fixes for kernel race conditions are often published with a hand-drawn ASCII diagram of the bad interleaving, citing a recent rt_spin_unlock use-after-free fix and a recent jbd2 deadlock fix as examples, and argues that tooling able to produce a similar picture automatically would help.

To replace that manual process, the author built and released MAccConc, short for Memory Access Concurrency: three tools, published on GitHub, for exploring thread interleavings in Linux kernel test cases. One automatically tests every A-B-A interleaving of a test case; the other two are a terminal UI and a GUI for exploring interleavings by hand. The underlying kernel instrumentation is also meant to eventually support fuzzing for race conditions, though the userspace side of that has not been built yet.

Finding candidate races means tracing every thread's memory accesses and looking for pairs, at least one of them a write, that touch overlapping memory; the SKI paper that partly inspired this project calls these communication points. SKI collected that trace by patching QEMU's TCG mode and replayed different interleavings from identical VM snapshots. MAccConc instead instruments the Linux kernel directly with ASAN in its outline mode (the compiler flag asan-instrumentation-with-call-threshold=0, selected in the kernel by CONFIG_KASAN_OUTLINE), which calls a helper function on every memory access; the kernel patches also turn off ASAN's usual optimization that merges helper calls for consecutive accesses, so every access still triggers a callback. Doing this inside the kernel, the author argues, means it could eventually surface higher-level information such as lock acquire and release events, which has not been implemented yet, and would in theory let the tests run on bare-metal hardware instead of only inside a VM. There is a real gap, though: ASAN skips helper calls for ordinary stack accesses unless they could go out of bounds, and for globals by default too, so some races on stack-resident objects such as wait queues may go undetected. The author also considered TSAN, which is built for data races and would report access atomicity, but rejected it because a compiler cannot emit both ASAN and TSAN hooks at once; using TSAN alone would cost the kernel's ASAN-based use-after-free detection unless its checks were rerouted through the TSAN hooks or the compiler itself changed.

For recording the trace, the author reused KCOV, the mechanism Linux already uses to feed fuzzers basic-block coverage, rather than ftrace. The reasons given are a simpler in-memory trace format that could help recover data from a crashed VM, static always-on instrumentation with near-zero overhead when disabled, and the author's own impression that KCOV is built for higher-frequency events than ftrace. Some races involve background work outside the traced call stack, such as loopback network packet handling or RCU callbacks; KCOV can optionally collect coverage for such background subsystems, but the author says most of the relevant background work is not yet wired up to it, and that today it is used mainly for fuzzing subsystems handling incoming device data such as Bluetooth and USB. A draft patch adds this support for RCU callbacks, and the author expects extending it to other subsystems to be straightforward.

The project's central idea is a way to name one exact memory access so it can be found again on a different run of the same test case: a "count-augmented stack trace." At every level of the call stack, it records the address of the called function together with a count of how many earlier calls to that same function, in that same calling frame, should be skipped, which pins down one specific call and the access inside it. That works where identifying an access by data address would fail, because the data may sit in a freshly allocated object on each run, and where identifying it purely by instruction address would fail, because the access might sit inside a shared function such as memcpy() or spin_lock() called from many places. Producing this required KCOV to also report function entry and exit events, which needed a compiler-side change to LLVM's SanitizerCoverage; the author's patch for that landed in the LLVM 23.1.0 release, a few months before this post.

To force a chosen ordering, the author added a new ioctl, KCOV_SET_DI, through which userspace attaches wait or wake actions to specific count-augmented stack traces, working against a shared array of flags with a configurable spin-wait limit. DI_STACK_WAKE_PRE sets a flag before the access runs; DI_STACK_WAIT spin-waits for a flag before the access runs; DI_STACK_WAKE_POST sets a flag after the access runs. Separate ioctls let userspace toggle the flags directly. This supports two styles of injection. Constraint-style A-happens-before-B injection pairs a WAKE_POST in one thread with a WAIT in another and leaves the rest of the ordering undetermined; it is simpler to reason about but needs recorded timing data to show roughly what happened, tends to need more constraints than a full ordering, and is what the current GUI and terminal UI implement. A fully specified, context-switch-style ordering instead has userspace choose the exact points where control passes between threads, for example one thread running while another spin-waits, then handing off with WAKE_PRE and WAIT and later handing back; this is the approach behind the automatic A-B-A interleaving tester.

The author credits discussions with Ned Williamson, creator of the sockfuzzer project, which explored concurrency bugs with a custom scheduler able to reschedule at synchronization primitives, as the inspiration for this work, alongside the published SKI paper for the idea of tracing communication points. MAccConc's code and a README with install and usage instructions are on GitHub. The version of the post available here breaks off right after introducing a "Demo: automatic testing" section, before any results appear, so no bug counts, real-world race conditions found, or performance numbers are on record; the extracted text also does not name its author, give a publication date, or name an institution behind SKI.

Key facts

  • Google Project Zero published MAccConc, three tools on GitHub (an automatic A-B-A interleaving tester, a terminal UI, and a GUI), for reproducing thread-interleaving bugs in Linux kernel test cases.
  • The tools instrument the kernel with ASAN in outline mode (asan-instrumentation-with-call-threshold=0, CONFIG_KASAN_OUTLINE) and reuse KCOV, the kernel's existing coverage mechanism, rather than ftrace, to trace every memory access.
  • Its core innovation is the count-augmented stack trace, a callee address plus a skip count that names one exact memory access stably across runs without VM snapshots, unlike the earlier SKI project; the compiler support for it landed in LLVM 23.1.0.
  • A new ioctl, KCOV_SET_DI, lets userspace force a chosen thread ordering with wait and wake flags (DI_STACK_WAKE_PRE, DI_STACK_WAIT, DI_STACK_WAKE_POST) attached to specific memory accesses, in either a loose A-happens-before-B mode or a fully specified context-switch-style ordering.
  • The project was inspired by discussions with Ned Williamson, creator of the sockfuzzer project; the version of the post available here breaks off before the demo section, so no bug counts or performance numbers are available, and the extracted text does not name its own author.

Why it matters

Race conditions are a genuine class of Linux kernel security bugs, and until now confirming or regression-testing one meant hand-editing the kernel with timing delays by trial and error, or hoping a fuzzer happened across the right interleaving. MAccConc turns that into something closer to a documented procedure: a stable name for one memory access across runs, plus an API to force threads into a chosen order around it. That is what makes an automated regression test, and eventually fuzzing, realistic for a bug class that has mostly resisted both. Building the instrumentation into the kernel itself, rather than a hypervisor as SKI did, is also what the author argues opens the door to richer scheduling information and to testing on real hardware instead of only inside a VM.

Who it affects

The people this is built for are Linux kernel developers and security researchers who hunt or fix concurrency bugs, along with maintainers of the KCOV, ASAN and KASAN tooling the project extends. It has no direct bearing on end users, on kernels other than Linux, or on userspace software; the DTrace-based technique the author mentions for macOS and Windows is background for comparison, not something MAccConc runs on.

How to use it

MAccConc's code and a README with install and usage instructions are on GitHub under that name. Using it means building a kernel with the relevant ASAN and KCOV configuration, including CONFIG_KASAN_OUTLINE, plus the KCOV_SET_DI ioctl support, which the author describes as living in a personal kernel branch rather than confirmed as merged into mainline Linux. Which of the three tools to reach for depends on the task: the automatic tester for exhaustively working through A-B-A interleavings, or the terminal and GUI tools for manually driving one specific interleaving with constraint-style delay injection.

How solid is it

The account is specific enough to check: exact compiler flags, exact ioctl action names, and a concrete LLVM release, 23.1.0, that shipped the SanitizerCoverage patch the whole scheme depends on. That is not the language of a rough plan. Against that, the version of the post available here cuts off right before the "Demo: automatic testing" section, so no bug counts, real race conditions found, or performance numbers back it up yet, and the caveats on record come from the author's own assessment rather than outside review, including that some on-stack races may go undetected and that the necessary kernel patches are only confirmed to exist in a personal branch.

Risks and caveats

By the author's own account, the ASAN-based approach can miss races on ordinary on-stack objects such as wait queues, because ASAN skips helper calls for plain stack accesses unless they could go out of bounds. Switching to TSAN would catch those through more accurate data-race detection, but the author says compilers cannot emit both ASAN and TSAN hooks at once, so using TSAN alone would cost the kernel's use-after-free detection unless the ASAN checks are rerouted through TSAN's hooks or the compiler itself changes. Coverage of background work such as loopback packet handling or RCU callbacks, needed to catch races involving it, is only partly wired up: a draft patch exists for RCU callbacks, but the author says most other relevant background work is not yet integrated. And because the extracted text stops right before the demo, none of it is backed yet by a results section, a bug count, or a benchmark.