01
Cache a successful computation.
Run the clean prompt and save activations at each candidate location. The clean output defines the behavior you want to recover.
A hands-on mechanistic interpretability experiment
Patch one activation. See what the model recovers.
Activation patching swaps a hidden state from a clean run into a corrupted run. If the original behavior returns, that location is causally implicated in the behavior under this intervention.
1. Clean runCache the hidden states that produce the target answer.
2. Corrupted runChange the causal feature and observe the broken behavior.
3. Patched runRestore one clean state and measure output recovery.
Find the causal handoff.
Each heatmap cell represents a synthetic intervention at one token position and transformer layer. Click a cell, change the patch width, or run the sweep to see how localization depends on the experiment.
Clean input
The Eiffel Tower is in
Corrupted input
The Colosseum is in
Clean targetParis 92%
Corrupted targetParis 18%
After patchParis 61%
little recoverystrong recovery
Clean target probability92%
Corrupted target probability18%
Patched target probability61%
Normalized restoration58%
The three-run protocol
Localization comes from an intervention, not from reading a representation in isolation. The comparison needs a clean baseline, a meaningful corruption, and a declared output metric.
02
Break the relevant feature.
Change the input so the model's target behavior degrades. A weak or off-target corruption gives an uninformative denominator.
03
Restore one state and measure recovery.
Insert a cached clean activation into the corrupted forward pass. Repeat across locations to build a causal-response map.
A bright cell is evidence, not a complete explanation.
Activation patching can identify places where a clean state is sufficient to restore output behavior in a specific setup. Interpretation still depends on corruption choice, metric, granularity, and interactions among components.
Localization is conditional.
The result describes this prompt family, intervention, component type, and score. Change those choices and the map can move.
Cost scales with the sweep.
Exact patching needs another forward pass for each tested location. A 12-layer, 8-token sweep means 96 patched runs, plus baselines.
98 runsRestoration is not necessity.
A patch showing sufficiency does not prove the component is uniquely necessary; redundant or interacting pathways may exist.
Read the claim carefully.
Choose the intervention that matches your question.
The site of the patch changes the unit of explanation. Broader patches can recover more behavior while localizing less precisely.
Read the primary work.
Towards Best Practices of Activation Patching in Language ModelsZhang and Nanda, 2023 - metrics, corruption choices, and methodological sensitivity
Interpretability in the Wild: A Circuit for Indirect Object IdentificationWang et al., 2022 - causal interventions in an end-to-end GPT-2 small circuit
Causal Analysis of Syntactic Agreement MechanismsFinlayson et al., ACL 2021 - causal mediation in pretrained language models
Use causal traces to form better questions, then test the whole system.
This lab uses a transparent synthetic response surface to teach the intervention protocol. Real model claims require real forward passes, held-out prompts, multiple corruption methods, and behavioral evaluation.