One means identical centered sample geometry under the linear kernel.
Compare representations, not coordinates.
Two neural layers can encode the same relationships in different bases. Centered Kernel Alignment compares their sample-to-sample geometry, revealing similarity that neuron-by-neuron inspection can miss.
Move the basis. Test the structure.
Layer B starts as a transformed copy of Layer A. Every metric below is calculated from the same deterministic sample matrices, so you can see which comparisons follow coordinates and which follow geometry.
Point geometry and centered Gram matrices
Matching sample IDs share colors. The lower heatmaps encode pairwise sample similarity after centering each representation.
Rotation · 45°
Whether CKA and local neighborhoods remain high together.
Direct flattened cosine after the selected transformation.
CKA assumes rows correspond to the same examples.
Similarity depends on what you choose to preserve.
CKA is useful because its invariances match common ambiguities in learned feature spaces. Those same invariances define what it cannot tell you.
Center each representation, then align its sample geometry.
Linear CKA normalizes the Hilbert-Schmidt Independence Criterion between feature matrices. In the linear case, it compares centered Gram structure without requiring equal feature counts.
CKA(X,Y) = ||YᵀX||²F / (||XᵀX||F ||YᵀY||F)Rows must mean the same samples.
Shuffle Layer B across examples and its internal geometry remains, but correspondence to Layer A is broken. CKA should fall.
Rotation is a basis change.
An orthogonal transform preserves dot products and Euclidean distances. Neuron axes move, yet representational geometry survives.
Isotropic scale cancels.
Multiplying every feature by the same constant changes raw magnitude but not normalized CKA. Unequal feature stretching can change the geometry.
One score is not a causal claim.
High CKA says two activation spaces share structure on these samples. It does not prove interchangeable functions, identical mechanisms, or equal downstream behavior.
Collect matched activations.
Rows represent the same examples in both models or layers. Columns are features and may differ in count. Choose samples broad enough to cover the behavior you want to compare.
Center across samples.
Subtract each feature mean, or equivalently center the Gram matrices. This removes global offsets so alignment reflects variation among examples.
Normalize shared geometry.
HSIC measures correspondence between centered similarities. Dividing by each representation’s self-similarity yields a bounded alignment score suitable for layer-by-layer maps.
Use the score as a map, not a verdict.
Representation comparisons become useful when tied to a controlled dataset, a hypothesis, and supporting behavioral evidence.
“A rotated representation can look different neuron by neuron while preserving every sample relationship.”
That is precisely the ambiguity linear CKA is designed to ignore. Coordinate-level correlation answers a different question.
Primary papers behind the metric.
These sources define HSIC, establish CKA for neural representation comparison, and demonstrate its use across architectures and training regimes.
Similarity of Neural Network Representations RevisitedKornblith et al., 2019 · introduces CKA as a representation-similarity measure with useful invariances
Feature Selection via Dependence MaximizationSong et al., 2012 · develops empirical HSIC estimators underlying kernel alignment
Do Wide and Deep Networks Learn the Same Things?Nguyen, Raghu, and Kornblith, 2021 · applies CKA across width and depth
Grounding Representation Similarity Through Statistical TestingDing, Denain, and Steinhardt, 2021 · examines reliability and statistical grounding of similarity measures
- ✓
Align examples, preprocessing, and aggregation before comparison.
- ✓
Report the kernel, centering, sample count, and layer shapes.
- ✓
Inspect uncertainty across sample subsets or bootstrap runs.
- ✓
Connect similarity maps to interventions and task behavior.