Learn · Intermediate

State Space Models and Mamba

A state space model is a way to process a sequence -- text, audio, DNA -- by carrying a fixed-size running summary forward one token at a time, updating it at each step and reading out an answer from it. That single design choice gives state space models their defining advantage over transformers: their cost grows only linearly with sequence length and their memory per token stays constant, which is why architectures like Mamba can chew through very long inputs cheaply. When you see a modern model described as 'hybrid' -- part attention, part something else -- that something else is usually a state space layer.

To see why this matters, recall the transformer's core cost. The attention mechanism lets every token look at every other token, which is powerful but expensive: for a sequence of length N, it does work proportional to N squared. Double the input and you quadruple the cost; a million-token context becomes brutally slow and memory-hungry, because the model also has to store a key-value cache that grows with every token. For years that quadratic wall was the price of admission for good sequence models.

State space models take a different route, borrowed from classical control theory and signal processing. Picture reading a book while keeping a single running summary in your head. You never re-read every previous page to understand the current sentence; you update your mental summary as you go. A state space model does exactly this: it maintains a hidden 'state' -- a fixed-size vector -- and at each token it applies a simple rule to blend the old state with the new input to produce a new state, then reads its output from that state. Because the state is a fixed size no matter how long the sequence is, the memory cost per token never grows, and the total compute is just linear in length. The mathematical form is a linear recurrence, and cleverly it can be computed either sequentially (cheap at inference, one token at a time) or in a parallel 'convolutional' form (fast during training) -- getting the best of both.

The catch, and the reason state space models took a while to work on language, is that a fixed, input-independent update rule is a blunt instrument. Early breakthroughs like S4 (Structured State Spaces, from Albert Gu, Karan Goel, and Christopher Re) cracked the numerical tricks needed to make these models stable and good at very long-range dependencies -- they crushed benchmarks on long sequences. But because the update was the same regardless of content, the model couldn't decide to pay special attention to an important word and ignore filler. It compressed everything uniformly, which is fine for signals but poor for language, where what to remember depends entirely on what you're reading.

Mamba, from Gu and Tri Dao, fixed this with one key idea: make the state update selective. Instead of fixed parameters, Mamba lets the update rule depend on the current input -- so the model can dynamically choose to let a token strongly modify the state (remember this) or barely touch it (skip this). That input-dependence is what a transformer's attention gets for free by comparing tokens, and giving it to a state space model is what finally made these architectures competitive with transformers on language modeling while keeping their linear-time, constant-memory efficiency. The trade-off is subtle: because everything the model 'knows' about the past is squeezed into that one fixed-size state, state space models can be worse at tasks requiring precise recall of an exact earlier token -- attention, which keeps every token available to look up, has an edge there.

That trade-off is why the winning pattern in 2026 is rarely pure state space and rarely pure attention, but hybrid. Models interleave many cheap state space layers with a handful of attention layers, getting most of the efficiency and long-context reach of state space models plus attention's sharp recall where it counts. You can see this directly in the news: Nvidia's Nemotron-Puzzle compression explicitly prunes both the mixture-of-experts and the Mamba layers of its model, which is a giveaway that the base architecture is a hybrid combining attention with state space layers. State space models didn't replace the transformer; they became the efficient backbone that lets long-context and hybrid models exist at all. If you understand 'carry a fixed-size summary forward, and let the input decide what to remember,' you understand the core of why they work -- and why the field keeps building them in.

Ground Truth. (2026, July 11). State Space Models and Mamba. Ground Truth. https://groundtruth.day/learn/state-space-models.html

@misc{groundtruth:state-space-models,
  title  = {State Space Models and Mamba},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/state-space-models.html}
}

Topics: state-space-models · mamba · architecture · long-context · efficiency