Data size drives grokking onset far more than model width

Neural networks trained past the point of memorizing their training data sometimes undergo a delayed shift to actually generalizing, a phenomenon known as grokking. A new paper maps where that transition happens across hyperparameter space rather than explaining why it happens, since the authors say prior theoretical work had already addressed the why. The mapping covers 384 configurations of two-hidden-layer multilayer perceptrons (MLPs) trained on modular arithmetic.

From those runs, the authors fit a power-law scaling relation for the generalization onset time, which they call T_grok. The fitted relation says T_grok scales with four hyperparameters: an exponent of -0.27 for model capacity (H, also called network width), -2.04 for the complexity of the training data (D), -0.50 for a hyperparameter denoted eta that the abstract does not otherwise define, and -0.64 for weight decay (lambda). Fit quality reaches an R-squared of 0.732 on its own and 0.821 once interaction terms between the hyperparameters are added.

The size of the exponents shows which factor matters most. Data complexity, at -2.04, is the dominant driver of the shift from memorization to generalization, not model capacity, at only -0.27. In practical terms, doubling the amount of training data speeds up generalization by about 4 times, while doubling the network's width speeds it up by only about 1.2 times.

The paper also reports a sharp phase boundary in weight decay: configurations with weight decay of roughly 1.0 or higher grok, and configurations below that threshold do not. Across the transition, the weight norm of the trained networks shrinks in a steady, monotonic way, a pattern the authors read as implicit regularization pushing the optimization toward low-complexity solutions.

The authors frame the result as a quantitative foundation for predicting and controlling when overparameterized networks shift from memorization to generalization, rather than only explaining why the shift happens. The underlying abstract reports only the fitted exponents and the R-squared values, not an absolute time scale, such as a number of training steps, for T_grok itself.

Key facts

The team fit a power-law scaling relation for the grokking onset time, T_grok, across 384 hyperparameter configurations of two-hidden-layer MLPs trained on modular arithmetic.
Data complexity has an exponent of -2.04 in the fitted relation versus -0.27 for model capacity: doubling the training data speeds up generalization about 4 times, doubling network width only about 1.2 times.
The power law reaches an R-squared of 0.732 alone and 0.821 once interaction terms between the hyperparameters are included.
A sharp phase boundary appears at a weight decay of roughly 1.0 or higher: configurations above that threshold grok, those below do not.
Weight norm compresses monotonically during the transition, which the authors read as implicit regularization pushing training toward low-complexity solutions.

Why it matters

Grokking already has a theoretical account of why it happens; this paper turns the open question of when it happens into a number. Fitting one power law across 384 configurations gives a way to predict how the onset of generalization shifts as four training choices change: model capacity, training-data complexity, the hyperparameter eta, and weight decay. The headline result, that data complexity dominates model capacity by a wide margin, about 4 times versus about 1.2 times the speedup from doubling each one, cuts against the assumption that scaling up model size is the main lever over training dynamics. Here the training data does most of the work instead.

Who it affects

The underlying abstract names no authors, institution, or publication date, so the result attaches to a research question rather than to any named lab or company. It speaks most directly to researchers studying grokking and generalization in overparameterized networks, and to anyone training networks on algorithmic or arithmetic-style tasks where memorization can precede generalization: the fitted scaling law gives them a way to estimate how changing data size, model width, or weight decay would move the point at which generalization kicks in.

How to use it

Within the tested setup, two-hidden-layer MLPs on modular arithmetic, the paper's own comparison is that doubling training data speeds up generalization about 4 times versus about 1.2 times for doubling network width, so spending a training budget on more or richer data looks like the more effective lever for reaching generalization sooner. The weight-decay phase boundary is a second lever: grokking occurred only in configurations with weight decay at or above roughly 1.0, so a configuration below that threshold in this setup would not be expected to grok at all.

How solid is it

The fit is reasonably strong on its own, an R-squared of 0.732 across the 384 configurations, and stronger once interaction terms between the hyperparameters are folded in, an R-squared of 0.821; the gap between the two suggests the hyperparameters do not act purely independently on the onset time. The scope is narrow by design: every configuration tested is a two-hidden-layer MLP trained on modular arithmetic, and the abstract does not describe testing any other architecture or task, so whether the same exponents, or even the same ranking of data over capacity, would hold elsewhere is untested by this paper.

Risks and caveats

The abstract leaves several things open. It does not define what the hyperparameter eta represents; only model capacity (H), training data (D) and weight decay (lambda) are glossed in the text. It gives no mechanistic explanation for why the exponents take the specific values they do, beyond the ranking that data outweighs capacity. And it reports no absolute time scale for T_grok itself, such as a number of training steps or wall-clock time, only the fitted exponents and the R-squared values. No authors, institution, or date appear in the abstract text either, so there is no way from the text alone to independently check the work's provenance.