ComPO aligns LLMs via comparison oracles, not gradients

Most direct preference alignment methods used to fit large language models to human preferences work by optimizing a differentiable loss over pairs of preferred and rejected responses, prized for being cheap on compute and memory. The paper argues that this setup runs into trouble when a preference pair has a small likelihood margin: the phenomenon known as likelihood displacement, where the model's probability mass shifts in ways that do not track the intended preference signal. The authors propose Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method built on comparison oracles rather than direct loss optimization. Instead of computing gradients of a preference loss on each pair, ComPO extracts directional information from the comparison itself, without directly optimizing a differentiable preference loss on the pairs.

The paper analyzes two variants. The basic offline scheme comes with a convergence guarantee, proved under three conditions: smoothness, gradient sparsity, and compatibility between the comparison oracle and a latent objective. Online ComPO keeps the same offline comparison mechanism but adds unlabeled policy generations, used for reverse-KL control relative to a reference policy, so the model is steered while being kept close to its starting point. Working from what the authors call the coverage perspective of preference fine-tuning, they also give a performance guarantee for a basic constrained scheme, this one resting on local coverage and in-distribution pairwise reward accuracy.

The method is tested on five model families: Mistral, Llama, Gemma-2, Qwen3, and Gemma-3. Across these, the authors report improvements over existing direct alignment methods, including on length-controlled win rates, and they present pair-level diagnostics whose evidence is consistent with mitigating likelihood displacement. The abstract does not give specific win-rate percentages or effect sizes for these improvements, and it does not say which of them were largest or by how much.

Key facts

ComPO is a zeroth-order alignment method that extracts directional information from preference pairs via comparison oracles, without directly optimizing a differentiable preference loss on them
It is aimed at likelihood displacement, a failure mode that shows up on preference pairs with small likelihood margins
The basic offline scheme has a convergence guarantee proved under smoothness, gradient sparsity, and oracle-to-objective compatibility
Online ComPO adds reverse-KL control against a reference policy, driven by unlabeled policy generations
Tested on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3, ComPO improves on existing direct alignment methods, including length-controlled win rates, though the abstract gives no specific percentages

Why it matters

Direct preference optimization and its relatives are the workhorse for aligning LLMs because they skip a separate reward model and train straight on preference pairs. The paper's starting point is that this shortcut has a cost: when a pair's likelihood margin is small, these methods can suffer likelihood displacement, where probability mass moves in a direction the preference signal did not actually call for. ComPO's pitch is to sidestep the problem at the root by not fitting a differentiable preference loss to the pairs at all, instead pulling a directional signal out of a comparison oracle, with convergence guarantees attached to the basic scheme.

Who it affects

The audience is researchers and engineers building or fine-tuning LLM alignment pipelines, specifically anyone currently relying on direct preference methods such as DPO-style training and running into likelihood displacement on pairs with small margins. The five families used in the experiments, Mistral, Llama, Gemma-2, Qwen3, and Gemma-3, indicate the method was checked across a broad, mostly open-weight model landscape rather than a single architecture.

How to use it

ComPO comes in two forms. The offline scheme is the base version, with the convergence guarantee. Online ComPO extends it by feeding in unlabeled generations from the policy being trained, using them for reverse-KL control against a reference policy, which keeps updates from drifting too far from the starting model. There is also a constrained variant, analyzed with a performance guarantee under local coverage and in-distribution pairwise reward accuracy, which is where the coverage perspective on preference fine-tuning comes in.

How solid is it

The claims are backed on two levels. Theoretically, the paper proves a convergence guarantee for the offline scheme under stated conditions, smoothness, gradient sparsity, and compatibility between the oracle and a latent objective, and a separate performance guarantee for the constrained scheme under local coverage and in-distribution pairwise reward accuracy. Empirically, experiments across Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 show improvements over existing direct alignment methods, including on length-controlled win rates, with pair-level diagnostics the authors say are consistent with mitigating likelihood displacement. The abstract does not give specific win-rate percentages or effect sizes, so how large these gains are cannot be judged from the text alone.

Risks and caveats

Both guarantees are conditional, not unconditional: the offline convergence result needs smoothness, gradient sparsity, and oracle-to-latent-objective compatibility to hold, and the constrained-scheme guarantee needs local coverage and in-distribution pairwise reward accuracy. Whether real training setups satisfy these assumptions is a separate question from whether the theorems are correct. On the empirical side, the pair-level diagnostics are described as consistent with mitigating likelihood displacement, which is evidence for the mechanism rather than a direct measurement of it, and the abstract does not specify which of the reported improvements were largest or by how much.

“ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them.”

— from the paper's abstract