Current dynamics
Linearized startAt initialization both predictions begin near zero.
Train the network. Watch the features move.
At infinite width, gradient descent can resemble kernel regression with a nearly frozen neural tangent kernel. Shrink the network or push updates harder, and learned features begin to drift away from that linearized prediction.
Θ(x,x′)=∇θf(x)·∇θf(x′)wide + small updates → kernel regime
Training samples
Toggle observations to alter the learned function.
Training progress
Feature drift
Kernel change
Function evolution
Compare the target, finite network, and frozen-kernel prediction.
Epoch 0
Interpretation
Features nearly frozenLarge width and small parameter movement keep the tangent features stable.
Train loss1.00
Network–NTK gap0.00
Feature drift0%
Active samples6
Linearized dynamics can predict nonlinear networks.
The tangent kernel measures how parameter changes co-move outputs across inputs. In a sufficiently wide network, that kernel changes little during training, turning gradient descent into a tractable kernel evolution.
The Jacobian becomes a feature map.
Each input is represented by its gradient with respect to parameters. Their inner products define which outputs learn together under gradient descent.
Width suppresses movement.
Under standard scaling, individual parameters move less as width grows.
width → ∞
Lazy is descriptive, not insulting.
Features can stay close to initialization while output fitting still succeeds.
Finite networks can learn features.
Narrower models or larger updates may alter the tangent kernel and improve representations.
The limit is not the whole network.
NTK predictions are asymptotic and parameterization-dependent, not universal behavior.
Measure the regime.
Record width, parameterization, initialization scale, optimizer, learning rate, training time, Jacobian drift, kernel drift, and finite-width error.
Compare function trajectories
Overlay finite-network predictions with kernel gradient flow throughout training, not only at convergence.
Measure tangent-kernel drift
Recompute the empirical NTK at checkpoints and report normalized change from initialization.
Sweep width and update size
A real regime claim should strengthen with width and weaken under aggressive feature movement.
Two useful limits, one finite model.
Hover or focus each regime to expose its assumptions.
Primary sources behind the lab.
The curves are illustrative. These papers introduce the neural tangent kernel, infinite-width correspondence, finite-width corrections, and feature-learning regimes.
NTK DEFINITION
Neural Tangent Kernel: Convergence and Generalization
Introduces the NTK and constant-kernel infinite-width dynamics.
INFINITE WIDTHWide Neural Networks of Any Depth Evolve as Linear Models
Connects wide-network training to first-order Taylor dynamics.
FINITE WIDTHOn Exact Computation with an Infinitely Wide Neural Net
Develops practical exact kernel computation for wide architectures.
FEATURE LEARNINGFeature Learning in Infinite-Width Neural Networks
Studies parameterizations where feature learning persists at infinite width.
Export a regime audit.
Preserve the width, scaling, kernel drift, and trajectory comparison. Then see how Superpowers builds inspectable tools and caches repeatable computer-use evaluations.
0 of 4 ready