Glonce

Inco AI's DFlash 2 lifts LLM decoding output 16-25%

infrastructureThe Glonce editorsHacker News🔥68Glonce rating68Impact70Novelty55Relevance85Surprise45Players65Credibility82
Inco AI's DFlash 2 lifts LLM decoding output 16-25%

Inco AI has released DFlash 2, an upgrade to its DFlash speculative-decoding drafter, saying agents consume tokens at a rate ordinary chat never approached and that every one of those tokens still costs a full forward pass through the model. DFlash itself is barely seven months old but, by Inco AI's account, already widely deployed: the company released it in January, and it now runs in the inference engines SGLang, vLLM, TensorRT-LLM, and llama.cpp. NVIDIA has measured up to 15 times the throughput using DFlash on its Blackwell GPUs, and Google has reported 3 times more tokens per second on TPUs, though neither company states what baseline those multiples are measured against. NVIDIA, Red Hat, and Modal, which earlier argued in its own post that speculative decoding is the single optimization that matters most for low-latency serving, have each published their own DFlash drafters. Meta, Poolside, Xiaomi, and NVIDIA each ship an official DFlash-based drafter alongside their own models (Muse Glimmer, Laguna, MiMo-V2.5-Pro, and Nemotron 3.5 Lightning, respectively). CoreWeave runs DFlash by default on its production Kimi K2.7 Code endpoint, which it says ranks fastest for that model on the Artificial Analysis benchmark. Across all of that adoption, DFlash models have been downloaded more than 3.5 million times on Hugging Face as of August 2026.

Speculative decoding is the technique behind all of this. Instead of generating one token per full, expensive pass through the large target model, a small draft model first guesses a block of upcoming tokens, typically 4 to 16 tokens long, and the target model checks the whole block in a single pass. Correct guesses turn one pass into several accepted tokens; wrong ones are simply discarded and decoding falls back to normal. For years, the draft model itself still worked one token at a time, autoregressively. DFlash's original contribution was to predict every position in the block in one parallel pass instead.

DFlash 2 pushes that further. Two additions drive the gain. Inco AI says the result is over 20% more accepted output from every verification pass than the original DFlash, for about 1% added draft-verify cycle latency (a more precise combined figure given later in the post is 1.3%), with the target model's output left provably unchanged. Across the benchmarks Inco AI tested, the gain ranges from 16% to 25%, averaging 1.05 extra accepted tokens per pass, or 21%, over the original DFlash; DFlash 2 also beats a DSpark-style comparison drafter by 0.48 tokens on average. The two additions target two separate sources of headroom Inco AI identifies in parallel, independent-position drafting: picking the best combination among already-good candidate tokens, and stopping accuracy from decaying toward the end of each drafted block.

The first addition is a lightweight path selector, aimed at a problem recent methods such as Domino and DSpark solve with a slower, sequential correction step that rewrites each position's full token distribution again. Inco AI argues that step is unnecessary. DFlash already keeps a list of the top 16 candidate tokens at each position in the block, and at the very first position, DFlash's single top-ranked guess is correct 85.4% of the time, while the correct token appears somewhere in that top-16 list 99.5% of the time. Inco AI calculates that an oracle able to always pick the right candidate from those 16 options would lift the average number of accepted tokens per verification pass from DFlash's actual 4.27 up to 6.79, and treats that gap as pure selection headroom. DFlash 2's selector scores every adjacent pair of candidates in one parallel pass, combining DFlash's own confidence in a token with a learned compatibility score between 256-dimensional embeddings of the two tokens, gated by context; Inco AI describes this as a form of low-rank bilinear attention over neighboring candidates. Only the final step, walking the precomputed scores to pick an actual path by greedy selection, sampling, or rejection sampling that preserves the target model's exact output distribution, runs sequentially. On its own, the selector already beats a DSpark-style correction module, using roughly 40 times fewer parameters and 16 times less latency overhead to do it; Inco AI notes the oracle's ceiling of 6.79 accepted tokens per pass remains above what the selector reaches, leaving room to improve further.

The second addition targets what Inco AI calls suffix decay: both DFlash's real accuracy and the oracle's own ceiling decline toward the end of a block. Even with perfect selection, the oracle's chance of having the right token on its candidate list falls from 99.5% at the first position to 87.8% at the last, which Inco AI attributes to the pool of correct candidates itself narrowing late in the block, not to any failure of selection. Comparing DFlash backbones with 3, 5, and 15 layers, the company found they perform almost identically at the first position but diverge further down the block, suggesting more depth helps. But depth is a blunt fix: ten extra transformer layers add capacity everywhere, including at early positions that did not need it, and add 15.2% to draft-verify cycle latency, eating into the efficiency that makes DFlash useful in the first place. Looking at where DFlash's attention actually goes, Inco AI found the share spent modeling dependencies within the block itself falls from 30% in the first layer to 8% in the fifth. So instead of adding more layers, DFlash 2 adds a small, dedicated module for that local, within-block work: a two-tap, content-adaptive depthwise convolution, reaching the current position and the one immediately before it, inserted before and after every attention and feed-forward sublayer. It adds only 16.5 million parameters, about 3%, to the five-layer DFlash model, bringing it close to the 15-layer version's performance while adding just 0.7% to cycle latency, against 15.2% for the ten extra layers. With the convolution in place, the within-block attention share across the last two layers falls further still, to 0.5% from 9.4%, which Inco AI reads as the convolution taking over the local work the extra attention had been doing.

Put together, Inco AI reports that DFlash 2, selector and convolution combined, leads every benchmark it tested: an average of 1.05 more accepted tokens per pass than the original DFlash (21%) and 0.48 more than the DSpark-style comparison, for a combined 1.3% added cycle latency. On the MATH-500 benchmark, the gap is visible position by position: DFlash 2 holds close to 86% accuracy all the way to the last position in the block, while every baseline it tested ends the block 6 to 9 percentage points lower.

Inco AI is releasing two DFlash 2 drafters alongside the writeup: one for Qwen3.8-27B and one for Meta's Muse Glimmer. Against Qwen3.8-27B's own native MTP path and a community DSpark drafter, and against Muse Glimmer's official DFlash drafter and a community DSpark drafter, the company says DFlash 2 beats both models' existing options, and beats the DSpark comparisons by more than a full token on average on both models. In practice, it says the new Qwen3.8-27B drafter lets SGLang serve at 2.7 to 3.4 times the throughput of plain autoregressive decoding at batch size 1, and the Muse Glimmer drafter reaches 3.1 to 4.6 times; full task- and concurrency-level breakdowns are on the respective model cards. Both drafters run in SGLang, vLLM, TensorRT-LLM, and llama.cpp; for Apple-silicon setups, Inco AI also walks through a configuration using a prebuilt build it calls oMLX, downloading the incoai/Qwen3.8-27B-DFlash2 drafter and enabling it on mlx-community/Qwen3.8-27B-4bit with draft quantization on, a runtime block size of 5, and verify mode set to dflash. The post states no price or licence terms for the drafters beyond their being published on Hugging Face. Inco AI describes DFlash 2 as the first piece of a larger, end-to-end inference-serving stack it is building, and says it will build custom drafters for other models, including private fine-tunes, on request for teams serving agents at scale.

Key facts

  • Inco AI released DFlash 2, adding a lightweight path selector and a local convolution to its DFlash speculative-decoding drafter; across benchmarks the gain runs 16-25% (21% on average) in accepted tokens per verification pass, for about 1.3% added draft-verify latency, with the target model's output left provably unchanged.
  • The selector picks the best path through DFlash's own top-16 candidate tokens at each position; an oracle that always chose correctly from that list would lift the average accepted tokens per pass from DFlash's actual 4.27 to 6.79, and the selector alone beats a DSpark-style correction module with roughly 40 times fewer parameters and 16 times less latency overhead.
  • A second addition, a two-tap local convolution, counters suffix decay, the drop in accuracy toward the end of each drafted block; it adds only 16.5 million parameters (3%) and 0.7% latency to the five-layer DFlash model, close to matching a 15-layer version that would otherwise cost 15.2% more latency.
  • Two new drafters are out on Hugging Face: one for Qwen3.8-27B, letting SGLang serve at 2.7 to 3.4 times the throughput of autoregressive decoding at batch size 1, and one for Meta's Muse Glimmer, reaching 3.1 to 4.6 times; both beat a community DSpark-style drafter by more than a full token on average.
  • The original DFlash, released in January, already runs in SGLang, vLLM, TensorRT-LLM, and llama.cpp, ships as the official drafter for models from Meta, Poolside, Xiaomi, and NVIDIA, runs by default on CoreWeave's production Kimi K2.7 Code endpoint, and has been downloaded more than 3.5 million times on Hugging Face as of August 2026.

Why it matters

DFlash's adoption already spans multiple serving engines, two cloud vendors' own throughput tests, a production endpoint a third-party benchmark ranks fastest for its model, and drafters that four model makers ship as official parts of their own releases. Speculative decoding looks close to a default layer in serving today's models, not a niche optimization. That is the backdrop DFlash 2 lands on. Inco AI's own framing is that agents make the underlying problem urgent: a single agent task can burn a chatbot-scale token budget in an afternoon by the company's account, and every one of those tokens still costs a full pass through the model. A technique that recovers over 20% more accepted tokens per pass, at roughly 1% added cost and with output the post says is left provably unchanged, is a close-to-free win for anyone already running DFlash or a similar drafter. It also signals a direction for speculative-decoding work generally: squeezing more out of the tokens a small draft model already proposes, and fixing where in the block its guesses get weaker, instead of paying for a slower, sequential correction step.

Who it affects

Most directly, anyone already serving models through SGLang, vLLM, TensorRT-LLM, or llama.cpp with a DFlash drafter, including, by Inco AI's account, NVIDIA and Google's own throughput testing, CoreWeave's production Kimi K2.7 Code endpoint, and the DFlash-based drafters that Meta, Poolside, Xiaomi, and NVIDIA already ship for their own models. Two model communities get a first-day DFlash 2 drafter outright: users of Qwen3.8-27B and of Meta's Muse Glimmer. More broadly, it affects anyone running agent workloads at meaningful scale, since that is the use case Inco AI designs for and the one where decoding cost dominates. Inco AI says it will build a custom drafter for other models, including private fine-tunes, on request, which extends the pool of who could be affected beyond the two models released alongside this post.

How to use it

DFlash 2 runs in SGLang, vLLM, TensorRT-LLM, and llama.cpp, and Inco AI has published the two new drafters, including incoai/Qwen3.8-27B-DFlash2, on Hugging Face. For Apple-silicon setups, the post walks through a specific configuration: install a prebuilt build Inco AI calls oMLX with DFlash 2 support, download the drafter, then in the Model Manager enable DFlash on the target model (mlx-community/Qwen3.8-27B-4bit in the example) with draft quantization turned on, a runtime block size of 5, and verify mode set to dflash. The post states no price or licence terms for the drafters beyond their being published on Hugging Face. Teams serving agents at scale, or wanting a drafter for a model Inco AI has not covered, including a private fine-tune, are invited to contact the company directly.

How solid is it

Every figure here is Inco AI's own: the company says it trained the DFlash and DSpark comparison drafters itself under matched conditions, while the MTP baseline is the model vendor's own shipped path, and the post cites no outside benchmark or replication for the DFlash 2 numbers specifically. There is some outside signal behind the broader approach, though: DFlash 1's throughput claims include NVIDIA's own Blackwell testing and Google's own TPU testing, not only Inco AI's numbers, and CoreWeave runs it in a production endpoint the post says is independently ranked fastest for its model on Artificial Analysis. The 'output provably unchanged' claim rests on a rejection-sampling step in the selector's final choice, which the post says restores the target model's exact output distribution. What is actually new in this release, the selector and the convolution, is supported only by Inco AI's own ablations and benchmark comparisons, including the MATH-500 result, published the same day as the two new drafters, so there is no outside confirmation yet of the DFlash 2-specific numbers.

Risks and caveats

The post gives no pricing or licence terms for the DFlash 2 drafters beyond publishing them on Hugging Face. It never states a calendar year for DFlash's original January release or for the drafters released alongside this post; a year is inferable only indirectly, from the 'as of August 2026' download count and the 'seven months' elapsed-time claim, not stated outright. The baselines behind two of the biggest numbers, NVIDIA's up to 15 times on Blackwell and Google's 3 times on TPUs, are not specified, unlike the explicitly autoregressive-decoding baseline used for the newer 2.7 to 4.6 times figures, so those are not directly comparable to each other. No individual researcher is named or quoted anywhere in the post; every claim is made in Inco AI's collective voice. And because DFlash 2 and its two drafters were released alongside this post, there has been no time yet for outside users to confirm the numbers independently, the way DFlash 1's adoption has partly done for the underlying approach.

“An agent writes in an afternoon what a chatbot writes in a month, and decoding sits under every one of those tokens.”

— Inco AI, in the DFlash 2 announcement