Glonce

Forward-Pass-Only training adapts LLMs without a backward pass

researchThe Glonce editorsarXiv🔥75Glonce rating75Impact65Novelty78Relevance100Surprise62Players25Credibility88

Researchers introduce Forward-Pass-Only MLP training (FPO), a way to adapt large language models to a domain without running a backward pass through the model body. Standard fine-tuning propagates an error signal layer by layer via backpropagation, which is what makes it memory-hungry and slow; FPO skips that. Against standard fine-tuning, FPO reaches 2.7 to 3.2 times the throughput while using about 40% less peak training memory, and it leaves off-domain benchmarks within seed-noise of the baseline, a property the authors say full-network fine-tuning does not reliably reproduce.

The method rests on one empirical observation: at the late layers of a transformer, the error in the model's output-layer prediction approximates the true gradient reasonably well, with a cosine similarity of 0.47 to 0.59 measured across six public models the authors surveyed. Building on that, the authors introduce a two-minute diagnostic that quantifies this approximation layer by layer for any model, which tells you where late-layer adaptation is viable before you commit to training. FPO itself then computes a single error signal at the output and applies it directly to each target layer: no signal is passed between layers, and no autograd graph is constructed at any point in training.

The authors evaluate FPO on three model families: OLMo-2-7B, Qwen3-8B, and Falcon3-7B. Across all three, FPO improves in-domain perplexity while MMLU, ARC-Challenge, HellaSwag, and Winogrande scores stay within seed-noise of the baseline model. The authors also test localizing standard supervised fine-tuning to the same target layers FPO uses, which they find is feasible for reaching the same regime, but it costs 2.2 times FPO's wall-clock time.

Key facts

  • FPO fine-tunes LLMs without a backward pass through the model body, reaching 2.7 to 3.2x the throughput of standard fine-tuning at about 40% less peak training memory.
  • It relies on the finding that at late transformer layers, the output-layer prediction error approximates the true gradient, with cosine similarity 0.47 to 0.59 measured across six public models surveyed.
  • A two-minute per-layer diagnostic identifies where this late-layer approximation is strong enough to make adaptation viable, and informs which layers FPO targets.
  • On three model families, OLMo-2-7B, Qwen3-8B, and Falcon3-7B, FPO improves in-domain perplexity while MMLU, ARC-Challenge, HellaSwag, and Winogrande stay within seed-noise of baseline.
  • Localizing standard supervised fine-tuning to FPO's same target layers can also reach this regime, but at 2.2x FPO's wall-clock cost.

Why it matters

Fine-tuning an open language model normally means backpropagating an error signal through every layer of the network, which drives up peak memory and slows training as models grow. FPO removes that cross-layer backward pass entirely, computing one error signal at the output and applying it straight to the target layers. The result is a domain-adaptation method that is faster and lighter on memory without the usual tradeoff of degrading the model's general-purpose skills.

Who it affects

The method targets anyone adapting open large language models to a specific domain under compute or memory constraints, such as teams running frequent fine-tuning passes on models like OLMo, Qwen, or Falcon and looking to cut training cost without giving up quality on the tasks the model already handles.

How to use it

FPO starts with a two-minute diagnostic, run once per model, that measures the cosine similarity between the output-layer prediction error and the true gradient at each layer; this identifies which layers are good candidates for late-layer adaptation. FPO then computes a single error signal at the output and applies it directly to each target layer, without propagating anything between layers and without building an autograd graph, which is what produces the throughput and memory gains.

How solid is it

The authors tested FPO on three model families, OLMo-2-7B, Qwen3-8B, and Falcon3-7B, and report in-domain perplexity improvement on all three, with MMLU, ARC-Challenge, HellaSwag, and Winogrande scores staying within seed-noise of the unmodified baseline. The core cosine-similarity approximation was checked across six public models. The abstract does not give the absolute perplexity or benchmark numbers behind these claims, only that the changes fall within seed-noise, and this is a preprint that does not name its authors or institutions in the text reviewed here.

Risks and caveats

The abstract does not specify a numeric tolerance for what counts as 'seed-noise', so how tight the off-domain equivalence is stays unclear without the full paper. The six models used for the cosine-similarity survey are not named, separately from the three families FPO was actually trained and evaluated on. The authors' own comparison shows that localizing standard fine-tuning to the same layers FPO targets can reach a similar regime, just at 2.2x the wall-clock cost, so FPO's contribution here is specifically an efficiency gain rather than access to an otherwise-unreachable outcome.