Glonce

MASS selects LLM fine-tuning data via manifold coverage

researchThe Glonce editorsarXiv🔥70Glonce rating70Impact60Novelty72Relevance100Surprise50Players25Credibility82

As supervised fine-tuning (SFT) datasets for large language models keep growing, picking a smaller high-value subset from a large candidate pool matters for both training cost and downstream performance. The paper argues that existing data selection methods measure diversity directly in the original embedding space, where geometric metrics entangle dominant semantic directions, fine-grained supervision differences, and local noise, which makes the resulting selection less reliable.

To address this, the authors frame data selection as a coarse-to-fine hierarchical coverage problem and propose a method called MASS. MASS first uses a dense autoencoder to learn low-dimensional principal manifold coordinates, grouping candidate examples coarsely by semantics. Within each group, it then performs quality-aware sparse feature coverage using a TopK sparse autoencoder, selecting examples on fine-grained features rather than on raw embedding geometry.

The authors test MASS on two datasets, Vision Flan and LLaVA-CoT. They report that MASS consistently outperforms strong data selection baselines across multiple selection budgets, and that in several settings it matches or even surpasses training on the full dataset while using only a small subset of the data. The text does not give the numeric results, the budget sizes compared, the names of the baseline methods, or who the authors are.

Key facts

  • MASS reframes data selection as a coarse-to-fine hierarchical coverage problem, addressing how raw embedding-space diversity metrics entangle dominant semantic direction, fine-grained supervision differences, and local noise.
  • A dense autoencoder learns low-dimensional principal manifold coordinates, used for coarse semantic grouping of candidate training examples.
  • Within each group, a TopK sparse autoencoder performs quality-aware sparse feature coverage to pick the final training examples.
  • On the Vision Flan and LLaVA-CoT datasets, MASS consistently outperforms strong data selection baselines across multiple selection budgets.
  • In several settings MASS matches or surpasses training on the full dataset while using only a small subset of the data; the paper does not state the exact figures.

Why it matters

Fine-tuning data pools keep growing, and training on all of it costs compute and time without necessarily improving quality. Prior data selection methods measure diversity straight in the raw embedding space, but that space entangles dominant semantic direction, fine-grained supervision differences, and local noise, which makes the resulting selection less reliable. MASS splits the problem into two clean steps, coarse semantic grouping followed by fine-grained sparse feature coverage, so each stage targets a different kind of structure instead of one metric trying to do both jobs at once.

Who it affects

The method targets researchers and engineers running supervised fine-tuning on large language models, particularly on vision-language tasks: both benchmarks used, Vision Flan and LLaVA-CoT, are multimodal instruction datasets. Anyone assembling a large SFT candidate pool and needing to cut cost by training on a smaller, well-chosen subset is the intended audience.

How to use it

MASS is applied as a two-stage pipeline: first train a dense autoencoder to learn low-dimensional principal manifold coordinates and use them to group candidate examples coarsely by semantics, then, within each group, train a TopK sparse autoencoder and use it to select examples through quality-aware sparse feature coverage. The text does not state a release, license, or code availability, so applying MASS currently means implementing the described pipeline rather than using a published tool.

How solid is it

The evidence is experimental, run on two datasets, Vision Flan and LLaVA-CoT, comparing MASS against unnamed strong data selection baselines across multiple selection budgets. The authors report consistent outperformance and, in several settings, matching or surpassing full-data training with a smaller subset. The source text does not give the actual accuracy numbers, the improvement margins, or the specific budget sizes involved, so the strength of the result cannot be judged from the abstract alone.

Risks and caveats

The claims rest on two datasets, both multimodal instruction sets, so it is unclear how MASS performs on plain-text SFT data or at other scales. The baselines it is compared against are not named, and neither are the paper's authors or their affiliation, which limits independent verification. Without concrete numbers, phrases like "consistently outperforms" and "matches or surpasses full data training" are the authors' own characterizations rather than figures a reader can check directly.