Divergence Decoding fuses specialist and generalist LLMs without retraining

Researchers have introduced Divergence Decoding, a training-free framework for combining a specialized language model with a general-purpose one at inference time, without fine-tuning either. The motivation is a known tradeoff: generalist LLMs reason well but often lack deep knowledge in specialized scientific domains, while domain-specific specialist models know the material but suffer what the authors call specialization side-effects, meaning weaker logical reasoning and reduced robustness. Divergence Decoding addresses this by repurposing the draft-and-verify structure used in speculative decoding into an adaptive routing mechanism between the two models. At each generated token, the method computes the Jensen-Shannon divergence between the specialist's and the generalist's output distributions to measure how much the two models disagree. When the specialist's prediction diverges significantly from the generalist's, the system treats that as a signal of a potential reasoning risk and instantly hands control of the next token to the generalist model. This lets the generalist's reasoning get injected dynamically into the specialist's output only where it is needed, while the specialist's domain expertise is preserved everywhere else, producing what the authors describe as inference-time policy composition of the two models. The method was tested across different model families, including Qwen and Llama series models, on scientific benchmarks considered challenging: GPQA, ChemBench, and ChemCoTBench. According to the results, Divergence Decoding outperformed both the domain-specialized and the general-purpose models it was built from, and it surpassed most single-model baselines. The authors present this as evidence that Divergence Decoding offers a general, training-free way to fuse different LLM capabilities through adaptive collaboration at inference time, rather than through additional training or fine-tuning.

Key facts

Divergence Decoding is a training-free method for fusing a specialist domain model with a generalist model at inference time.
It reworks the draft-and-verify structure of speculative decoding into a token-level routing mechanism between the two models.
Jensen-Shannon divergence between the two models' output distributions is measured at each token to detect disagreement.
When the specialist diverges significantly, control instantly routes to the generalist to reduce reasoning risk.
Tested on Qwen and Llama model families across GPQA, ChemBench, and ChemCoTBench, it outperformed both source models and most single-model baselines.

Why it matters

Specialist and generalist models have long been treated as a tradeoff: fine-tune a model on a narrow domain and it gains knowledge but tends to lose general reasoning sharpness and robustness, a pattern the paper calls specialization side-effects. Divergence Decoding proposes fusing the two at inference time instead of retraining either one, by letting a general model take over mid-generation exactly when the specialist's next-token prediction looks unreliable. That reframes capability fusion as a decoding-time control problem rather than a training problem.

Who it affects

The method targets teams building or deploying domain-specialized LLMs in scientific fields such as chemistry, where a fine-tuned specialist model is common but its weaker general reasoning can be a liability. It is also relevant to anyone working with speculative decoding infrastructure, since Divergence Decoding reuses that draft-and-verify pipeline shape for a different purpose: capability composition rather than pure speed.

How to use it

Divergence Decoding requires no fine-tuning or retraining of either model: it works directly with an existing specialist and an existing generalist model, run side by side, with routing decided per token by their distributional disagreement. The paper reports results specifically with Qwen and Llama family models, suggesting the technique is meant to be applied across open model families rather than tied to one architecture.

How solid is it

The claims rest on evaluation across multiple model families (Qwen and Llama) and three benchmarks framed as challenging for scientific reasoning: GPQA, ChemBench, and ChemCoTBench. The paper reports that Divergence Decoding beat both the specialist and generalist models it was composed from, and surpassed most single-model baselines, which is a reasonably broad test setup, though the source text does not give the specific score margins or list the exact baseline models compared.

Risks and caveats

The abstract does not specify numerical performance gaps, model sizes, or which single-model baselines were not surpassed, so the strength of the improvement cannot be quantified from the source alone. Reliance on Jensen-Shannon divergence as a proxy for reasoning risk is a design choice whose failure modes, such as false triggers where the specialist is correct but simply distributed differently from the generalist, are not addressed in the available text.