Glonce

Skill Training lifts LLM pass rate 8.1pp, keeps 85% after distillation

researchThe Glonce editorsarXiv🔥73Glonce rating73Impact68Novelty77Relevance100Surprise47Players22Credibility80

Researchers propose scaffold-mediated post-training, a method for closing the gap between post-training, which optimizes only a model's parameters, and inference-time procedural scaffolds, which are normally designed separately from that training. The authors argue this disconnect makes it difficult for a model to automatically acquire and internalize complex strategies, since the scaffold sits outside the parameters rather than becoming part of them. Their fix organizes procedural scaffolds into an evolvable graph structure that co-evolves with the model's parameters through three steps: discovery, distillation, and dynamic recompilation.

The authors instantiate this paradigm as 'Skill Training' and test it on a benchmark called FeatureBench. Skills the system discovers automatically lift the passed rate by 8.1 percentage points. After progressive distillation, the model still achieves a 27.7% passed rate with no external scaffold at all, which the authors report as a distillation retention rate of 85.2%, defined as the post-distillation passed rate divided by the with-skill passed rate. They report that this distilled model significantly outperforms a standard SFT (supervised fine-tuning) model trained on the same data.

Key facts

  • Scaffold-mediated post-training organizes inference-time procedural scaffolds into an evolvable graph that co-evolves with model parameters through discovery, distillation, and dynamic recompilation.
  • The authors instantiate the paradigm as 'Skill Training' and evaluate it on FeatureBench.
  • Automatically discovered skills lift the passed rate by 8.1 percentage points on FeatureBench.
  • After progressive distillation, the model reaches a 27.7% passed rate with no external scaffold, a distillation retention rate of 85.2% (post-distillation divided by with-skill passed rate).
  • The distilled model significantly outperforms standard SFT trained on the same data.

Why it matters

Post-training today tunes only a model's parameters, while the procedural scaffolds that guide it at inference time are built independently. The authors say this split makes it hard for a model to automatically pick up and internalize complex strategies: the know-how lives in the scaffold, not in the weights. Scaffold-mediated post-training closes that gap by letting scaffolds and parameters evolve together, so strategies a model relies on externally can be folded back into the model itself.

Who it affects

The work targets researchers and teams building post-training pipelines for large language models, particularly those relying on external procedural scaffolds to get models to perform complex, multi-step tasks. It speaks to anyone weighing whether such scaffolding can be distilled into the base model rather than shipped alongside it at inference time.

How to use it

The method runs as a cycle: discovery of useful procedural scaffolds, distillation of what they teach into the model's parameters, and dynamic recompilation of the scaffold graph as the model improves. The authors' instantiation, Skill Training, applies this cycle directly on FeatureBench, and the reported result is a model that keeps most of its scaffold-driven skill after the scaffold itself is removed. No pricing, licensing, or release details are given; this is a research method, not a shipped tool.

How solid is it

The result rests on a single benchmark, FeatureBench, with no description in the source of what it measures, its size, or its domain beyond the name itself. The source gives the size of the improvement (8.1 percentage points) but not the starting passed rate it improved from, and it names no specific model architecture or size used in the experiments. No author names or affiliating institutions are given in the visible text, so the work cannot yet be attributed to a specific lab.

Risks and caveats

Beyond the missing baseline and benchmark detail, the source states no timescale, compute cost, or training budget for the method, making it hard to judge how expensive scaffold-mediated post-training is to run in practice. The comparison to standard SFT is described only as the distilled model 'significantly outperforming' it, without a specific SFT passed rate to check the claim against.