Watch a batch become a stable activation.

Batch normalization uses the current mini-batch during training, updates running statistics for later, then switches behavior at evaluation. Change the batch, momentum, noise, and distribution shift to see the contract directly.

Run the simulatorRead primary sources

From raw activations to normalized outputs

Input activationsbatch μ 3.00 · σ² 2.25
Normalized outputsoutput μ 0.00 · σ² 1.00
y = (x − μbatch) / √(σ²batch + ε)Training uses this batch for normalization and then blends its statistics into the running estimate.
Current batch mean3.00

Average of this batch's raw activations.

Running mean0.30

Exponential estimate retained across training batches.

Statistic usedBatch

Train uses batch statistics; eval uses running statistics.

Output drift0.00

Distance between output mean and the affine target.

Measure locally

During training, each channel is standardized with statistics computed across the mini-batch. Small or unrepresentative batches make those estimates noisy, so the same example can receive a different output beside different neighbors.

Remember gradually

Running mean and variance are buffers, not gradient-trained parameters. Each training batch nudges them. Momentum controls how quickly the stored estimate follows a changing data distribution.

Freeze for inference

Evaluation uses the stored running statistics so one prediction does not depend on whichever other examples happen to share its batch. Forgetting to switch modes, or using stale statistics, changes model behavior.

Read the symptoms before changing the layer.

Tiny batches

Batch estimates become noisy. Consider synchronized statistics, gradient accumulation that preserves the true normalization batch, GroupNorm, or LayerNorm depending on the architecture.

Distribution shift

Frozen running statistics may no longer match production inputs. Audit preprocessing first, then evaluate recalibration or domain-specific normalization with held-out evidence.

Train/eval mismatch

Reproducible validation failures often come from the wrong mode, running buffers that were never updated, or checkpoint code that omitted non-parameter state.

Primary reading

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Ioffe & Szegedytorch.nn.BatchNorm1d PyTorchtf.keras.layers.BatchNormalization TensorFlow