FOAM, a lighter SOAP optimizer, matches SOAP's loss with half the state

FOAM, a lighter SOAP optimizer, matches SOAP's loss with half the state

Second-order optimizers such as Shampoo and SOAP are pitched as alternatives to Adam and AdamW that converge faster per step in transformer training. The authors of a new paper say they remain costly in practice: even SOAP carries substantial optimizer-state memory and eigendecomposition overhead. Their question is whether SOAP's benefits can be kept at nearly the efficiency of AdamW. They answer yes with FOAM, short for Factored One-sided Adam-Moment, a lightweight SOAP variant.

FOAM makes two changes. First, it uses input-only preconditioning. It drops SOAP's output-side Kronecker factor, which cuts memory and removes one eigendecomposition. The authors say the Kronecker structure of the Hessian shows this is exact when the output-side curvature is diagonal and nearly lossless when it is near-diagonal, and they report seeing this empirically.

Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator built on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, the authors say this estimator exactly recovers SOAP's dense second moment. They measure only a 15-20% violation of the K-FAC assumption, which they report has negligible effect on the update.

On results, the authors say their experiments support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of LLM pre-training. It uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters. The authors conclude that SOAP's gains need not require SOAP's cost. Code is available at github.com/naver-ai/foam.

Key facts

  • FOAM (Factored One-sided Adam-Moment) is a lightweight variant of the SOAP second-order optimizer.
  • It drops SOAP's output-side Kronecker factor, which reduces memory and removes one eigendecomposition.
  • It replaces SOAP's dense rotated second moment with a factored Adam-moment estimator; the measured K-FAC violation is 15-20%, with negligible effect on the update according to the authors.
  • The authors report validation loss matching SOAP and Muon baselines, half of SOAP's optimizer state, and Muon-class wall-clock time, from 124M to 760M parameters.
  • Reported robustness covers a three-fold learning-rate range; code is at github.com/naver-ai/foam.

Why it matters

Second-order optimizers promise faster per-step convergence than AdamW, but the authors say their memory and eigendecomposition costs keep them impractical, even for SOAP. FOAM is an attempt to keep SOAP's gains while cutting that cost, aiming at nearly the efficiency of AdamW. If the claim holds, the authors' own summary is that SOAP's gains need not require SOAP's cost.

Who it affects

Teams pretraining transformer language models and researchers working on optimizers are the audience. The reported results cover LLM pre-training at 124M to 760M parameters, so the evidence speaks most directly to models in that range.

How to use it

The authors released code at github.com/naver-ai/foam. The paper describes FOAM as a SOAP variant, so it is meant to be compared against SOAP and Muon in a pre-training setup. The text does not say which learning rates make up the three-fold range in which FOAM stays robust, so tuning guidance has to come from the repository.

How solid is it

The source is the paper's abstract on Hugging Face Papers, and every result is the authors' own report. The theory is conditional: the exactness claims hold for a diagonal output-side curvature, and the second-moment estimator recovers SOAP's dense second moment exactly only under the K-FAC Fisher assumption with a diagonal output curvature. The authors measure a 15-20% violation of that assumption and report negligible effect on the update. The text does not say FOAM beats SOAP or Muon; it says FOAM matches them in validation loss.

Risks and caveats

No concrete validation-loss numbers, perplexities or benchmark scores are given. No actual wall-clock speedup figures or memory figures in GB are given; "Muon-class wall-clock time" is the only timing statement. No results are reported above 760M parameters, and scaling beyond that range is not claimed. The text does not say how many training tokens, datasets or hardware were used. No peer review, publication venue or submission date is given.

“SOAP's gains need not require SOAP's cost”

— FOAM paper abstract