Current operation
Awaiting tokenStep through the sequence to update the hidden state.
Decide what the state remembers.
Mamba turns fixed state-space dynamics into content-dependent ones. Replay a selective scan to see which tokens write strongly, which signals persist, and why linear sequence processing is not the same thing as attention.
hₜ = Āₜ hₜ₋₁ + B̄ₜ xₜBₜ, Cₜ, Δₜ depend on xₜ
INPUT STREAM
Remember the access code buried inside ordinary text.
Write gate
Retention
Read gate
State evolution
Four teaching channels summarize the recurrent state after each token.
t = 0
Interpretation
Content-dependent memorySelective parameters let the recurrence react differently to each token.
Signal retained0%
Distractor writes0
State width4
Pairwise scores0
Selection changes the recurrence, not the asymptotic scan.
A conventional linear state-space layer applies input-independent dynamics. Mamba makes key parameters functions of the current token, then uses a hardware-aware parallel scan to keep sequence length scaling linear.
Content can open the write gate.
A high-value token can enter state strongly while ordinary filler barely perturbs it. This is the lab’s simplified picture of input-dependent Δ, B, and C.
Linear in sequence length.
The recurrence carries a fixed-width state instead of materializing every query-key pair.
O(L)
State is compressed history.
Unlike attention, the model cannot revisit an arbitrary past token directly; information must survive in the state.
Parallel scan matters.
Associative scan structure and kernel fusion turn the recurrence into an efficient GPU implementation.
Long context is not perfect recall.
Linear scaling removes the quadratic matrix, but finite state and learned gates still create memory tradeoffs.
Test what the state forgets.
Sweep distance, distractor density, gate sharpness, and state width. Compare exact retrieval, throughput, memory, and failure under distribution shift.
Needle retrieval
Move the target earlier, add similar distractors, and measure whether the answer signal survives until the query.
Selective-copy stress
Require the model to copy only marked tokens. Fixed recurrence should reveal why content-dependent updates matter.
Attention comparison
Report matched parameter count, hardware, batch size, context length, and quality. Big-O notation is not a benchmark.
Three mechanisms, three different promises.
Hover or focus each mechanism to expand its operational contract.
Primary sources behind the scan.
The state values are illustrative teaching signals. These papers define structured state spaces, Mamba’s selective mechanism, hardware-aware scan, and the state-space duality developed in Mamba-2.
STRUCTURED STATE SPACES
Efficiently Modeling Long Sequences with Structured State Spaces
Introduces S4 and the structured recurrence foundation.
SELECTIVE STATE SPACESMamba: Linear-Time Sequence Modeling with Selective State Spaces
Defines input-dependent selection and the hardware-aware parallel algorithm.
PEER-REVIEWED RECORDMamba at ICLR 2024
OpenReview record for the conference paper and discussion.
STATE SPACE DUALITYTransformers are SSMs: Generalized Models and Efficient Algorithms
Develops structured state-space duality and Mamba-2.
Export a memory audit.
Turn each slider setting into a reproducible evaluation note. Then see how Superpowers builds inspectable agent tools and caches repeated computer-use evaluations.
0 of 4 ready