No explicit shrinkage in this comparison.
AdamW decouples weight decay from the gradient that Adam rescales. Compare plain Adam, an L2 penalty folded into the adaptive gradient, and decoupled AdamW on the same synthetic parameter.
Compare update pathsFor SGD, L2 regularization and multiplicative weight decay can align under common assumptions. Adaptive optimizers transform coordinates using moment estimates, so adding λw to the gradient also sends the penalty through that adaptive scaling. AdamW applies decay directly to parameters instead.
No explicit shrinkage in this comparison.
The penalty enters the adaptively scaled gradient.
Decay is applied directly to the parameter.
Cumulative absolute change from decay alone.
Adam tracks first and second moments, then divides the momentum estimate by a root-mean-square scale. Coordinates with different gradient histories receive different effective step sizes.
update = m̂ / (√v̂ + ε)When λw is added to the loss gradient, Adam’s moment estimates and denominator also act on the regularization term. The resulting shrinkage is coordinate- and history-dependent.
g ← ∇L(w) + λwAdamW performs the adaptive gradient update and a direct parameter decay as distinct operations. Learning-rate and decay schedules still interact, but the decay term is no longer normalized by Adam’s gradient statistics.
w ← w − η·AdamUpdate − η·λw