Glonce

BlockLock makes LLM inference reproducible without custom kernels

researchThe Glonce editorsarXiv🔥58Glonce rating58Impact45Novelty60Relevance100Surprise25Players10Credibility70

Reproducible LLM inference means a request produces the same output regardless of which other requests share its batch. The paper behind this story starts from the cost of existing ways to get that. Batch-invariant kernels constrain every operator in the model. The authors' own CoRun-style baseline isolates prefill and uses fixed-shape decode, which gives up shared prefill compute and pads decode.

The authors present BlockLock as a cheaper route. It keeps shared batched prefill and pins only the numerical path of the protected request, by fixing the batch geometry that the request sees. It comes in two modes. BlockLock-Rows uses fixed-width rows and keeps batched decode of all requests. BlockLock-Pack uses terminal-anchored packing with an isolated dummy segment, then extracts the protected request's KV state for target-only decode. Under a stated per-deployment contract, both modes preserve the target's output bitwise when its companions change.

The evaluation covers five dense models: Qwen3-8B and 14B, Llama-3.1-8B-Instruct, and Gemma-3-4B and 12B-IT. These ran on three single-GPU deployments (NVIDIA A6000, A100 and H100), with tinyBenchmarks and BFCL as the benchmarks. The authors also report initial tensor-parallel experiments on two NVIDIA A6000 GPUs. In the reported tests, both modes preserve target sequences and Top20-logit bits.

On speed, the headline numbers are for BlockLock-Rows. At equal delivered work, it reduces latency by 53.5% to 55.0% and improves throughput by 115.2% to 122.2% relative to vLLM Batch-Invariant. The authors add that, compared with batch-invariant kernels, BlockLock needs no custom kernel implementations and constrains only the protected requests.

Key facts

  • BlockLock keeps shared batched prefill and fixes only the batch geometry seen by the protected request, so its output stays bitwise identical when other requests in the batch change.
  • Two modes: BlockLock-Rows (fixed-width rows, batched decode of all requests) and BlockLock-Pack (terminal-anchored packing with an isolated dummy segment, then target-only decode).
  • At equal delivered work, BlockLock-Rows cuts latency by 53.5% to 55.0% and raises throughput by 115.2% to 122.2% relative to vLLM Batch-Invariant.
  • Tested on five dense models (Qwen3-8B/14B, Llama-3.1-8B-Instruct, Gemma-3-4B/12B-IT) and three single-GPU setups: NVIDIA A6000, A100 and H100.
  • Unlike batch-invariant kernels, it needs no custom kernel implementations and constrains only the protected requests.

Why it matters

Getting the same answer from a model no matter who else is in the batch is a hard requirement for reproducible inference. The paper argues that the two existing approaches both cost a lot: batch-invariant kernels constrain every operator, and isolating prefill with fixed-shape decode gives up shared prefill compute and pads decode. BlockLock tries to pay only for the request that needs protection. The reported gain over vLLM Batch-Invariant is a 53.5% to 55.0% latency reduction and a 115.2% to 122.2% throughput improvement for BlockLock-Rows, at equal delivered work.

Who it affects

The paper is aimed at people who serve LLMs with batched requests and need a particular request to give bit-identical output whatever its batch companions are. The evaluation covers Qwen3, Llama-3.1 and Gemma-3 models from 4B to 14B parameters on NVIDIA A6000, A100 and H100 GPUs, so those are the setups the results speak to.

How to use it

The abstract describes the method but gives no usage instructions. In outline: choose BlockLock-Rows if you want fixed-width rows and batched decode of all requests, or BlockLock-Pack if you want the protected request's KV state extracted for target-only decode after terminal-anchored packing with an isolated dummy segment. The authors say neither mode needs custom kernel implementations. The bitwise guarantee holds under a stated per-deployment contract.

How solid is it

This is an arXiv preprint, and the numbers are the authors' own. The tests span five dense models, three single-GPU deployments, and the tinyBenchmarks and BFCL benchmarks. Both modes preserved target sequences and Top20-logit bits in the reported tests. The latency and throughput figures are for BlockLock-Rows only, measured against vLLM Batch-Invariant. The abstract does not say which models, GPUs or benchmark gave the low or high end of the reported ranges.

Risks and caveats

The bitwise guarantee is conditional on a stated per-deployment contract whose terms the abstract does not give. The tensor-parallel experiments, on two NVIDIA A6000 GPUs, are described only as initial. The latency and throughput gains are stated for BlockLock-Rows only; no figures are given for BlockLock-Pack, or against the authors' CoRun-style baseline. The evaluated models are all dense, and the claim of preserved bits is limited to the reported tests.

“Reproducible LLM inference requires that a request produces the same output regardless of the other requests in the batch.”

— BlockLock paper abstract, arXiv 2611.06977