Two clocks run at different speeds.
Example fitting can happen quickly; reorganizing weights into a compact, rule-like solution may take far longer.
First it memorizes.
Then it understands.
Train a synthetic rule learner past zero training error. Change data coverage, capacity, regularization, and time to see why test accuracy can remain flat before rising abruptly.
memorization → compression → delayed generalization
Accuracy through training
Training, held-out performance, and compression proxyFITTING
Step 0
Train loss2.400
Generalization gap1%
Compression2%
Onset estimate8.1k
Zero loss is not the end of learning.
In grokking experiments, an overparameterized model first fits the training set while held-out performance stays near chance. Continued optimization plus an implicit or explicit simplicity pressure can later favor a more general algorithm.
Regularization changes which solution wins.
Weight decay can gradually penalize brittle memorizing solutions, but too much prevents fitting at all.
Coverage matters.
More examples constrain the rule and usually shorten the delayed-generalization phase.
Capacity cuts both ways.
Extra parameters make memorization easy while also providing room for an algorithmic representation.
A sharp jump needs context.
Always report the task, split, optimizer, parameterization, regularization, and full training horizon.
Separate fitting from learning the rule.
Training accuracy alone cannot distinguish a lookup table from an algorithm. Track held-out accuracy, weight norms, representation structure, and sensitivity across seeds for long enough to observe the delayed phase.
Stopping at zero training loss erases the phenomenon by definition.
Look for a window between underfitting and unconstrained memorization.
Onset time can vary widely, so a single dramatic curve is weak evidence.
Primary sources behind the lab.
The simulator is a teaching model, not a reproduced training run. These papers establish the original observation and later mechanistic or theoretical accounts.
ORIGINAL OBSERVATION
Grokking: Generalization Beyond Overfitting
Introduces delayed generalization on algorithmic datasets.
PHASE TRANSITIONSTowards Understanding Grokking
Studies training dynamics, representation, and phase changes.
MECHANISTIC ACCOUNTProgress Measures for Grokking via Mechanistic Interpretability
Tracks circuit formation before generalization becomes visible.
THEORYDeep Networks Always Grok and Here Is Why
Connects late generalization to optimization and implicit bias.
Export a grokking audit.
Preserve configuration and the predicted phase before comparing runs or claiming emergence.