Glonce

virtio-nvgpu gives KVM guests near-native NVIDIA GPU access

infrastructureThe Glonce editorsHacker News🔥66Glonce rating66Impact60Novelty72Relevance70Surprise55Players25Credibility80
virtio-nvgpu gives KVM guests near-native NVIDIA GPU access

virtio-nvgpu is an open-source project that lets a Linux KVM guest talk almost directly to a physical NVIDIA GPU. Instead of translating graphics API calls between guest and host, as virtio-gpu's Venus does, it forwards NVIDIA kernel driver ioctls at the driver ABI level: the guest runs NVIDIA's own user-mode drivers unmodified, the same Vulkan, OpenGL, CUDA and NVENC libraries, building GPU command buffers locally and talking to the same physical card the host owns. The target use case is headless GPU streaming, where a compositor inside a monitor-less VM renders, composites and encodes video on the GPU and only the compressed bitstream leaves the machine.

The project has been benchmarked on an RTX 3060 running driver 595.99.02, comparing a guest against the same host running bare metal with an identical headless Vulkan load. Above roughly 2 ms per frame, which covers every frame a real game draws, the guest lands within 2% of bare-metal performance; below that threshold the cost of waiting on the GPU starts to show against a frame that barely exists. On the CPU side, an unpaced load of about 100 fps for 12 seconds on a single guest produced 813,691 rendered frames against just 13,792 backend messages, roughly one boundary crossing per 59 frames, almost all of it device setup rather than per-frame work. A Wayland client presenting inside a guest, with NVENC capturing on the game's own device, produced 618 H.264 frames that ffmpeg decoded without error.

Four simultaneous guests were run on the same RTX 3060 under identical load, achieving 25.84, 26.49, 25.57 and 25.79 fps each (103.7 fps combined, versus 102.9 fps for a single guest alone), with near-identical p50 frame times of 39.165 to 39.168 ms. All four rendered correctly and encoded H.264 at once, each paced at exactly 60 Hz, with no NVENC session limit reached; the project notes four is what it tested, not a ceiling it found, and eight guests has not been tried. An RTX A2000 on driver 615.71.09 also renders but has not been benchmarked. Supported NVIDIA driver ABI profiles are 535.129.03, 580.178.04 and 595.71.05, matched by version range; anything older than the first is refused rather than guessed at, since forwarding an ioctl whose layout has never been seen risks a plausible wrong answer instead of an error.

The project explains why it exists by contrasting itself with three alternatives. Venus-style API translation serializes every Vulkan or OpenGL call: games issue 1,000 to 5,000 draw calls per frame, and with a 16.6 ms budget at 60 fps, 1 to 3 ms of serialization can eat 6 to 18% of that budget before any GPU work happens; it also burns host CPU and cannot support guest-side encoding because GPU buffers stay host-owned. The DRM native context approach used for Intel and AMD does not exist for NVIDIA. VFIO passthrough gives native performance but dedicates the whole GPU to one VM, which rules it out for multi-tenant setups. virtio-nvgpu instead crosses the VM boundary only per ioctl, about 5 to 20 messages a frame versus roughly 2,000 for Venus, because a render loop issues none: submission is a write to memory NVIDIA's driver has already mapped. At very light frames, what remains is not forwarding overhead but the cost of waking a sleeping guest, about 0.02 ms.

The codebase splits into a GPL guest kernel driver (needed to touch kernel symbols) and a permissively licensed, VMM-agnostic host device crate, following the layout of chromeos/virtio-media; a VMM adopts it by implementing a small set of traits. A planned sandboxed 'isolate' helper process, meant to hold device file descriptors per guest and issue ioctls unprivileged, has not been built yet; today the backend holds those descriptors itself inside the VMM's process. CUDA is forwarded but untested beyond enumeration, and the project states its own benchmark numbers do not support any comparison against another hypervisor, because none was run.

Key facts

  • virtio-nvgpu forwards NVIDIA kernel driver ioctls at the ABI level instead of translating graphics API calls, letting a KVM guest run NVIDIA's unmodified user-mode drivers for Vulkan, OpenGL, CUDA and NVENC.
  • On an RTX 3060 (driver 595.99.02), a guest lands within 2% of bare-metal render performance above about 2 ms per frame, and a headless Wayland/NVENC pipeline produced 618 H.264 frames ffmpeg decoded without error.
  • Four simultaneous guests on one RTX 3060 rendered and encoded H.264 correctly at once (25.84 to 26.49 fps each, 103.7 fps combined versus 102.9 fps for one guest alone), though the project calls this what it tested, not a found limit.
  • Supported NVIDIA driver ABI profiles are 535.129.03, 580.178.04 and 595.71.05; any driver older than the first is refused rather than guessed at to avoid misforwarding an unrecognized ioctl layout.
  • CUDA support is forwarded but untested beyond enumeration, the planned sandboxed 'isolate' process is unbuilt, and the project states its numbers do not support any comparison against other hypervisors since none was tested.

Why it matters

Sharing a single NVIDIA GPU across several VMs has meant either VFIO passthrough, which hands the whole card to one guest, or virtio-gpu's Venus, which translates every graphics API call and eats CPU and frame budget doing it. virtio-nvgpu instead forwards driver-level ioctls, so the guest's own NVIDIA drivers do the real work locally and the VM boundary is crossed only for a handful of messages per frame rather than roughly 2,000 under Venus. That combination, near-native performance plus support for multiple simultaneous guests on one card, is aimed squarely at headless GPU streaming setups.

Who it affects

Operators building headless GPU streaming or multi-tenant GPU virtualization on KVM, developers working on VMM GPU device models, and anyone currently choosing between VFIO's single-tenant passthrough and Venus's API-translation overhead for NVIDIA hardware specifically, since a native DRM context path like Intel and AMD have does not exist for NVIDIA.

How to use it

The project is open source, split into a GPL guest kernel driver (required to touch kernel symbols) and a permissively licensed, VMM-agnostic host device crate, following the repository layout of chromeos/virtio-media. A VMM adopts the device by implementing a small set of traits, such as descriptor chains as Read/Write, an event queue and guest/host memory mapping, without patching the crate; optional capabilities degrade rather than failing to build. Full benchmark method and raw runs are documented separately in the project's BENCHMARKS.md.

How solid is it

Performance numbers come from a single card and driver combination, an RTX 3060 on driver 595.99.02, under one synthetic Vulkan and NVENC load; an RTX A2000 on a different driver has been shown to render but was not benchmarked. Four simultaneous guests were tested and shared the card evenly, but the project is explicit that this is what it ran, not a discovered ceiling, and that its numbers do not support comparison against any other hypervisor since none was tested.

Risks and caveats

NVIDIA's kernel driver ABI is not stable across releases, so only explicit driver version ranges are supported and anything older than the oldest profile is refused outright. CUDA interop is forwarded but untested beyond enumeration, the planned sandboxed 'isolate' helper process that would hold device descriptors per guest has not been built (the backend currently holds them itself inside the VMM's process), and testing has not gone beyond four guests or workloads heavier than vkcube at 720p.

“Four is what was run, not a limit found.”

— virtio-nvgpu project documentation