Adapting CPython 3.15.1 for the yk JIT: faster on all but one AWFY benchmark

The author of yk, a framework that adds a meta-tracing JIT compiler to a lightly modified C interpreter, set out to answer a question put to him at the end of October. He had just shown data where yklua, the yk-based Lua, ran 2× faster than the PUC Lua interpreter it is built on, on the team's benchmark suite. Someone asked: "how do you know if this works for large interpreters?" PUC Lua is at least an order of magnitude smaller than CPython or CRuby. The team had looked at CPython about a year earlier but yk lacked several things CPython needed. A quick LLM experiment suggested it was worth another look.

The outcome is ykcpython, CPython adapted for yk. The author says it works and brings meaningful gains over normal CPython and over Python's experimental JIT. He puts the effort at 3-4 days of work on ykcpython directly, plus a separate week and a half of generic yk bug fixing (missing atomic loads, unimplemented bit widths in sub, stackmap fixes, a fork-related bug, and an incompatibility where a CPython test wiped an environment variable that was read afterwards) before he could even look at ykcpython itself. In total he added just under 450 lines of code.

The headline numbers use the AWFY benchmark suite (the team's fork of it). The comparison has three builds: cpython is normal CPython 3.15.1; cpython_jit is CPython 3.15.1 with the experimental JIT turned on; ykcpython is commit 31b4d387. Times are in seconds, lower is better, with means and 99% confidence intervals bootstrapped from 30 process executions. They were measured on a 64 core AMD EPYC 7773X Debian machine in performance mode, and the post links the repeatable experiment and raw data. The author's summary: ykcpython is faster than cpython on every benchmark except DeltaBlue, where it is about 2.5% slower, and faster than cpython_jit on 8 out of 13 benchmarks.

The author is blunt about limits. The post is not meant to suggest ykcpython is in any way ready for people to use; it definitely is not. Nor should anyone expect the same performance on a wider set of programs. His view is that benchmark suite performance is a hoop to jump through rather than an end in itself. The real aim is to show what adapting an interpreter for yk is like: the stages, and how the 450 lines were enough.

A refresher on yk: given a lightly modified C interpreter, it outputs a VM with a meta-tracing JIT. It does not aim to be the fastest possible JIT; it aims to make the language implementation you know somewhat faster while retaining 100% compatibility. It finds hot loops in the guest language (Lua or Python, say), traces the C code executed while interpreting, optimises the traces and compiles them to machine code. The interpreter must mark potential loops with a YkLocation struct. Optional optimisation hints, such as saying that a seemingly dynamic value almost never changes, are needed for good performance. Placing YkLocations is relatively easy; placing hints needs deep knowledge of the interpreter and careful measurement. A simple interpreter may need fewer than 10 changed lines for a basic JIT. yk dislikes "threaded" interpreters that use goto between opcodes, since they give it too few chances to intercede.

The author's method when he does not know an interpreter well is to start with the minimum changes that get some traces recorded and compiled, and to keep the language's test suite passing from the first step, which makes unsound optimisations less likely later.

For CPython he first adapted what he later learned is called the "tier 1" interpreter. The diff was about 200 lines, mostly small one-offs: switch CPython to its built-in but non-default switch-based interpreter, and make CPython's C stack resizing account for yk's shadow stack. The key addition was YkLocations at backwards jump opcodes. This first version produced correct traces and passed the test suite, albeit using more memory than he liked, but it was much slower than he expected. The cause was CPython's quickening (called "specialisation", added 4 years ago), which rewrites generic opcodes like "add objects" into specialised ones like "add two ints" and back again. In the simple yk integration, each transition threw away the associated trace and created a new one, so traces were rarely used long enough to repay their creation cost.

There are techniques to get around this, and he briefly tried them. Then he remembered a conversation with some CPython developers the year before and confirmed that CPython has a "tier 2" interpreter, an alternative implementation of core parts of the main interpreter loop, mostly meant for CPython's experimental JIT. A very quick experiment suggested it was a better fit for yk. He removed the tier 1 support and added yk support to tier 2 in a mere 80 lines. The core piece walks the tier 2 executor's trace and, for each _JUMP_TO_TOP micro-op, creates a yk location for the jump target if there is none yet.

The build setup: cpython is configured with clang-21, --enable-optimizations and --with-lto. The same LLVM version is used for cpython and cpython_jit because the JIT requires LLVM-21. ykcpython uses ykllvm, based on an LLVM about 6 months later than LLVM-21, which might slightly distort comparisons, though the author expects the effect to be fairly minor. ykcpython is configured with --without-computed-gotos, --without-tail-call-interp, --enable-experimental-jit=interpreter and --without-ensurepip.

The metric is wall-clock time from start to finish. The author calls it the least worst single metric, and it includes start-up and warm-up, so it deliberately tends to favour normal interpreters and penalise JIT compilers. To offset that somewhat, each AWFY benchmark runs for 10 in-process iterations. Timings later in the post come from running time once as a normal user, without being careful that the machine was idle, because during early optimisation the effects are generally large and waiting an hour per change is impractical. The post then begins a section on optimising interpreter dispatch, starting from the Mandelbrot benchmark on cpython, and the available text ends there.

Key facts

  • ykcpython, CPython 3.15.1 adapted for the yk meta-tracing JIT, is faster than normal CPython on every AWFY benchmark except DeltaBlue, where it is about 2.5% slower.
  • It is faster than CPython's experimental JIT on 8 out of 13 benchmarks; means and 99% confidence intervals come from 30 process executions on a 64 core AMD EPYC 7773X machine.
  • The work took 3-4 days directly on ykcpython plus a separate week and a half of generic yk bug fixing, and added just under 450 lines of code.
  • A tier 1 interpreter port (about 200 lines) ran much slower than expected because CPython's quickening kept discarding traces; switching to the tier 2 interpreter needed only 80 lines of yk support.
  • The author says ykcpython is definitely not ready for use and that results on a wider set of programs will not match these numbers.

Why it matters

yk had shown a 2× speedup on yklua versus the PUC Lua interpreter it is based on, but PUC Lua is at least an order of magnitude smaller than CPython or CRuby. The open question was whether the approach scales to a large, heavily optimised interpreter. This post is the author's answer: on CPython 3.15.1, a modest patch of just under 450 lines gives a working JIT that beats stock CPython on all but one AWFY benchmark and CPython's own experimental JIT on 8 of 13. The notable detail is how little code that took, and that the key move was reusing CPython's existing tier 2 interpreter, where yk support was only 80 lines.

Who it affects

Mainly people who build or maintain language interpreters written in C and want a JIT without writing one from scratch, and people watching how Python's experimental JIT compares with alternative approaches. Ordinary Python users are not affected yet: the author says ykcpython is in no way ready for people to use.

How to use it

ykcpython itself is not for use. What the post offers is a method for adapting an interpreter to yk. Start with the minimum changes that get some traces recorded and compiled, which means understanding the build system and placing semi-sensible YkLocations (here, at backwards jump opcodes). Keep the language's test suite passing from the first step. Avoid threaded goto-style dispatch, which yk dislikes; ykcpython is built with --without-computed-gotos and --without-tail-call-interp. Expect to hit generic yk bugs first. Add optimisation hints only later, with careful measurement. The post also links the repeatable experiment and raw data for the main table.

How solid is it

The main table is reasonably rigorous: 30 process executions per benchmark, 99% confidence intervals, an idle 64 core machine in performance mode, and a custom benchmark runner. The post links a repeatable experiment and raw data. The author chose wall-clock time, which includes start-up and warm-up and so tends to favour normal interpreters over JIT compilers, and ran 10 in-process iterations per benchmark. The timings after the main table are single runs of time as a normal user, without checking that the machine was idle. This is the author's own report on a suite his team forked. The text reviewed does not include the per-benchmark values or the later optimisation steps.

Risks and caveats

The author says results on a wider set of programs will not be as good as the table, and that benchmark performance is a hoop to jump through rather than an end in itself. ykllvm is based on an LLVM about 6 months newer than the LLVM-21 used for the two CPython builds, which might slightly distort the comparison. The first tier 1 port also used more memory than the author liked. ykcpython is slower than cpython on DeltaBlue (about 2.5%) and slower than cpython_jit on 5 of the 13 benchmarks. The 3-4 days figure excludes the week and a half of generic yk fixing. No release or upstreaming plan is given in the text.

“My view is that benchmark suite performance is a hoop to jump through rather than an end in itself: a system which benchmarks well might perform poorly in the real world; but a system which benchmarks poorly nearly always performs poorly in the real world.”

— The post's author