Glonce

Chain-of-Models finds the best LLM bias auditor differs by bias

researchThe Glonce editorsarXiv🔥74Glonce rating74Impact68Novelty72Relevance98Surprise52Players58Credibility78

LLMs are increasingly used as automated judges, but their verdicts remain vulnerable to cognitive biases. Existing fixes, prompt-based debiasing or human review, either fail to generalize across bias types or do not scale to the volume of judgments modern pipelines produce. Researchers propose Chain-of-Models (CoM), an audit pipeline in which a second model inspects the first model's reasoning trace before the final judgment is produced. Their key design question: should the auditor be the same model, a model from the same family, or a model from a different family entirely.

The team evaluated the approach across 9 LLMs drawn from 6 model families, tested against 4 cognitive biases and 4 factual datasets. Two findings stood out. First, a model's own resistance to bias when working alone does not predict how well it performs as an auditor of another model: Kimi-K2.5 was the strongest standalone model on several biases, yet turned out to be a weak auditor for Qwen2.5-72B's biased reasoning traces. Second, the best auditor is bias-specific: GPT-4o was strongest at catching bandwagon effects, appeals to authority, and distraction, while GLM-5 was strongest specifically at catching sycophancy.

From these findings the researchers built a per-bias auditor-selection rule that, given a bias type, scores candidate auditors on functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a held-out calibration/test split, the selector reached 0.884 accuracy across the four biased slices, ahead of 0.824 for the strongest single fixed auditor and 0.805 for a no-audit baseline. The researchers say they have released the data, configurations, and an LLM-agent skill implementing the approach at an anonymously hosted code repository, anonymous.4open.science/r/chain-of-models-B585.

Key facts

  • Chain-of-Models (CoM) has a second LLM audit the first model's reasoning trace before a judgment is finalized, targeting cognitive bias in LLM-as-judge setups.
  • Tested across 9 models from 6 model families, against 4 cognitive biases and 4 factual datasets.
  • Standalone bias resistance does not predict audit skill: Kimi-K2.5 is a strong standalone model but a weak auditor for Qwen2.5-72B's biased traces.
  • The best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction; GLM-5 is strongest on sycophancy.
  • A per-bias auditor-selection rule reaches 0.884 accuracy on a calibration/test split, versus 0.824 for the best fixed auditor and 0.805 for no audit.

Why it matters

LLM judges now grade other models' outputs in evals, RLHF-style pipelines, and agent tool chains, and a judge that inherits or falls for a cognitive bias quietly corrupts every downstream decision that trusts it. Prompt-based debiasing patches one bias at a time and breaks on the next; human review does not scale to the volume of judgments these pipelines produce. This work reframes the problem: instead of building one bias-proof judge, add a second model whose only job is to audit the first model's reasoning before the verdict is final. The findings show that choosing that auditor is a real design decision, not an afterthought, since auditor skill differs sharply by bias.

Who it affects

Teams building LLM-as-judge pipelines for evaluation, benchmark grading, RLHF, or agent output verification, anywhere a model's verdict is trusted without a human in the loop. It also concerns anyone using Kimi-K2.5, Qwen2.5-72B, GPT-4o, or GLM-5 as a judge or auditor in such a pipeline, since the paper names concrete strengths and weaknesses for each.

How to use it

The researchers released the data, configurations, and an LLM-agent skill implementing the per-bias auditor-selection rule at an anonymously hosted repository, anonymous.4open.science/r/chain-of-models-B585. The selection rule takes a bias type as input, then scores candidate auditor models on functional diversity, per-bias standalone resistance, and calibrated audit effectiveness to pick the best available auditor for that specific bias, rather than relying on one fixed auditor for every case.

How solid is it

The claims rest on a study spanning 9 models across 6 model families, tested against 4 cognitive biases and 4 factual datasets, with results checked on a held-out calibration/test split rather than only in-sample. The reported gap, 0.884 accuracy for the per-bias selector against 0.824 for the best fixed auditor and 0.805 for no audit, is a measured improvement rather than a dramatic leap. The abstract names no authors, institution, or publication venue, and the code repository is hosted anonymously, typical of an early or under-review arXiv preprint but a limit on independently verifying provenance.

Risks and caveats

The paper does not describe the mechanism by which the auditor model actually inspects the first model's reasoning trace: no prompt, protocol, or interface is specified beyond the word 'inspects', making it hard to judge how the method generalizes beyond the tested models and biases. No timeline is given for production use of the auditor-selection rule, and the anonymous hosting of the code means the released artifacts cannot yet be tied to a verified source.