Glonce

IDQD-RDO lets video codecs optimize directly for perceptual quality metrics like MS-SSIM and LPIPS

researchThe Glonce editorsHugging Face🔥76Glonce rating76Impact72Novelty72Relevance72Surprise45Players18Credibility82
IDQD-RDO lets video codecs optimize directly for perceptual quality metrics like MS-SSIM and LPIPS

Block-based video codecs like VVC pick their coding parameters by balancing rate against distortion, and the conventional distortion measure is the sum of squared errors (SSE) because SSE splits cleanly into independent block-wise sums, which makes per-block optimization easy. Full-reference image quality assessment (FR-IQA) metrics such as MS-SSIM or LPIPS often track human visual perception more closely than SSE, but until now they could not be used inside the encoding loop: they do not decompose block by block, and they typically need the fully decoded image as input before they can be evaluated at all. Researchers propose a method called IDQD-RDO that gets around this. It approximates a broad class of FR-IQA metrics with an input-dependent quadratic distortion (IDQD), a quadratic form whose matrix comes from the Hessian of the target metric evaluated at the source video. To make that distortion computable one block at a time, the authors introduce two ways to simplify the Hessian: keeping only its block-diagonal part, or keeping only its diagonal. For both simplifications they build estimators that need nothing more than matrix-vector products with the Hessian, obtained through automatic differentiation, so no full Hessian matrix has to be formed or stored. The method was tested across five FR-IQA metrics on the Kodak and CLIC image datasets inside the VVC codec. IDQD-RDO produced 14.2-36.7% BD-rate savings measured under the target metric, all without any changes to the decoder, at a cost of 10-30% extra encoding complexity.

Key facts

  • IDQD-RDO approximates full-reference quality metrics (MS-SSIM, LPIPS, and others) with an input-dependent quadratic distortion derived from the metric's Hessian at the source video.
  • Two Hessian approximations make the distortion computable block by block: keeping the block-diagonal, or keeping only the diagonal.
  • Both approximations are estimated using only matrix-vector products with the Hessian obtained via automatic differentiation, avoiding the need to build the full Hessian.
  • Tested with five FR-IQA metrics on Kodak and CLIC inside the VVC codec, IDQD-RDO achieves 14.2-36.7% BD-rate savings under the target metric with no decoder changes, at a cost of 10-30% extra encoding complexity.

Why it matters

Video codecs have long been stuck optimizing for pixel-level error (SSE) because it is the only distortion measure that splits neatly across blocks, even though perceptual metrics like MS-SSIM and LPIPS match human judgment of quality better. IDQD-RDO is a way to bring those perceptual metrics into the actual encoding decision, not just use them afterward to measure quality, by turning them into a quadratic approximation that behaves like SSE well enough to plug into standard block-based rate-distortion optimization.

Who it affects

The method is aimed at video codec researchers and engineers working on encoder-side optimization for standards such as VVC, and by extension anyone building or tuning video compression pipelines who wants encoders to target perceptual quality rather than raw pixel error.

How to use it

IDQD-RDO requires no changes to the decoder: it only changes how the encoder scores candidate coding decisions during rate-distortion optimization. Applying it means computing a quadratic approximation of the chosen FR-IQA metric's Hessian at the source video, using one of the two proposed simplifications (block-diagonal or diagonal), via Hessian-vector products obtained through automatic differentiation. The tradeoff is a 10-30% increase in encoding complexity compared with conventional SSE-based RDO.

How solid is it

The account here rests on the paper's own abstract, which describes experiments across five FR-IQA metrics on the Kodak and CLIC datasets using the VVC codec, reporting 14.2-36.7% BD-rate savings and 10-30% encoding overhead. The source text does not name the paper's authors, their institutions, a publication date, which five metrics beyond MS-SSIM and LPIPS were tested, or any baseline method other than conventional SSE-based RDO, so those details cannot be reported here.

Risks and caveats

The reported gains are measured under the target metric itself, on two benchmark datasets (Kodak and CLIC) and one codec (VVC), so it is not established how the approach performs on other content, other codecs, or against metrics beyond the five tested. The method also carries a real cost: 10-30% more encoding complexity than conventional SSE-based RDO, which matters for any deployment sensitive to encoding time or compute budget.