Glonce

Alignment Forecasting paper predicts misalignment before fine-tuning, with a 5,000-question benchmark

researchThe Glonce editorsarXiv🔥76Glonce rating76Impact70Novelty78Relevance100Surprise60Players32Credibility82

Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned. The authors of a new arXiv paper say that inspecting the data at face value often does not settle whether this will happen, and that today the problem is caught only after training, by auditing the resulting model.

To complement those post-hoc audits, the paper introduces Alignment Forecasting: the task of predicting alignment failures before training. A forecaster is given a target model, a fine-tuning dataset and a failure mode such as deception or sycophancy. It outputs the probability that fine-tuning would meaningfully increase that failure mode.

To measure progress, the authors built ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions spanning 17 target models, 32 datasets and 16 failure modes. Frontier models prompted directly perform poorly on it.

The authors therefore propose a forecasting scaffold with two parts. First, an LLM reads the dataset and rates how strongly and broadly it pushes the model toward misbehavior. Second, a simple learned model combines that rating with the failure mode's base rate and the target model's prior tendency. According to the authors, this forecasts well above chance. It also beats a model fine-tuned on the task, and a simple forecaster that is allowed to see how weaker models behaved after fine-tuning on the same data.

The scaffold's signals also flag problematic training examples that a frontier-model classifier misses. Filtering those examples out of real post-training data such as UltraChat results in more aligned models on the authors' multiple-choice evaluation in most cases. The benefit in open-ended conversations is unclear.

The authors conclude that more progress is needed before forecasts can reliably guide training data curation in practice. Their results, they say, suggest that forecasting many alignment failures before training can be tractable in the SFT setting.

Key facts

  • Alignment Forecasting is the task of predicting, before training, the probability that fine-tuning a target model on a given dataset would meaningfully increase a failure mode such as deception or sycophancy.
  • ALIGNMENTFORECASTBENCH has over 5,000 forecasting questions across 17 target models, 32 datasets and 16 failure modes.
  • Frontier models prompted directly perform poorly; the proposed scaffold pairs an LLM's rating of a dataset with a simple learned model using the failure mode's base rate and the target model's prior tendency.
  • The scaffold forecasts well above chance and beats a model fine-tuned on the task and a forecaster that sees how weaker models behaved after fine-tuning on the same data.
  • Filtering flagged examples out of post-training data such as UltraChat gave more aligned models on the multiple-choice evaluation in most cases, with unclear benefit in open-ended conversations.

Why it matters

Misalignment from narrowly flawed training data is currently found only after training, by auditing the finished model. The paper frames a complementary step: estimate the risk before spending the compute. It also supplies a benchmark, ALIGNMENTFORECASTBENCH, so progress on the task can be measured rather than argued.

Who it affects

The work is aimed at people who fine-tune language models on datasets and at those who audit or curate post-training data. The paper's real-data test uses UltraChat, a post-training dataset.

How to use it

The source describes the scaffold rather than a ready tool. An LLM reads the dataset and rates how strongly and broadly it pushes the model toward misbehavior, and a simple learned model combines that rating with the failure mode's base rate and the target model's prior tendency. The abstract does not mention a release of code, data or the benchmark.

How solid is it

The benchmark is sizeable: over 5,000 questions, 17 target models, 32 datasets and 16 failure modes. The scaffold is reported to forecast well above chance and to beat two baselines. The source gives no specific accuracy, AUC or other score, and no size for the alignment improvement from filtering. This account rests on the paper's abstract, and the results are the authors' own.

Risks and caveats

The authors say more progress is needed before forecasts can reliably guide training data curation in practice. The filtering benefit showed up on their multiple-choice evaluation in most cases, while the benefit in open-ended conversations is unclear. The claim of tractability is limited to the SFT setting (supervised fine-tuning); nothing is claimed about other training regimes. Frontier models prompted directly do poorly, so the approach depends on the extra scaffold.

“More progress is needed before forecasts can reliably guide training data curation in practice, but our results suggest that forecasting many alignment failures before training can be tractable in the SFT setting.”

— Paper abstract, arXiv 2609.35805