Glonce

Data-DPO picks fine-tuning data by reading the target model's own feedback

researchThe Glonce editorsarXiv🔥68Glonce rating68Impact65Novelty60Relevance95Surprise50Players30Credibility80

Researchers propose Data-DPO, a data selection method for supervised fine-tuning that is built around the specific model being trained rather than a generic notion of sample quality. The authors argue that existing selection methods usually treat data value as a relatively static property and pay limited attention to how well a sample fits the capability distribution of the target model, so a sample that is valuable for one model may not be valuable for another. Data-DPO addresses this by running a one-step probe: it observes the target model's local training feedback on different candidate samples, then converts the differences in the model's internal activations across samples into pairwise data preferences. Those pairwise preferences train a lightweight reward model that learns which samples the target model itself prefers. In the final selection stage, Data-DPO combines three signals: the learned target-model preference, external quality scores, and marginal diversity among the selected samples, to build what the authors describe as a more stable and effective training subset. Tested on the Vision-Flan and LLaVA-CoT datasets, Data-DPO is reported to consistently outperform existing data selection baselines across multiple data budgets, and to stably surpass the performance of training on the full, unfiltered dataset.

Key facts

  • Data-DPO scores candidate SFT samples by probing the target model's own local training feedback through one-step probing, rather than using a fixed, model-independent notion of data value.
  • It converts activation differences among samples into pairwise data preferences and trains a lightweight reward model on those preferences.
  • Final data selection combines the learned target-model preference with external quality scores and marginal diversity among selected samples.
  • On the Vision-Flan and LLaVA-CoT benchmarks, Data-DPO outperforms existing data selection baselines across multiple data budgets.
  • Training on the Data-DPO-selected subset is reported to stably surpass training on the full dataset.

Why it matters

Most data selection methods for fine-tuning score samples once, as if a sample's value were a fixed property of the data itself. Data-DPO's premise is that value is relative to the model being trained: a sample that pushes one model's capabilities forward may do little for another. By probing the target model directly and turning its reaction to samples into a preference signal, the method ties selection to the specific model in training rather than to a generic quality score, which is a different design point from most prior selection work.

Who it affects

The method targets teams doing supervised fine-tuning, particularly of multimodal, instruction-following models: both benchmarks used, Vision-Flan and LLaVA-CoT, are vision-instruction and chain-of-thought datasets. Anyone fine-tuning under a fixed compute or data budget, where picking the most useful subset matters more than using all available data, is the intended audience.

How to use it

Data-DPO is presented as a selection stage that runs before or alongside fine-tuning: it probes the target model on candidate samples, trains a small reward model from the resulting preferences, then ranks candidates by combining that preference score with external quality scores and a diversity term before picking the training subset. The abstract does not state whether code, trained reward models, or the selected subsets have been released.

How solid is it

The claims come from experiments on two datasets, Vision-Flan and LLaVA-CoT, where the authors report consistent gains over existing data selection baselines across multiple data budgets, plus a result that beats full-data training. The abstract does not give the actual numbers behind those comparisons, so the size of the improvement cannot be assessed from the source text alone, and no author names or institutional affiliation are stated.

Risks and caveats

No specific numeric results, such as accuracy figures or percentage gains, are given in the available text, and the exact values used for the 'multiple data budgets' are not defined. Evaluation is limited to two datasets in the vision-instruction and chain-of-thought space, so how the method generalizes to other domains or to text-only SFT is untested in what is described here.