state-space-models
Causal masking and prefix invariance: how a model is stopped from reading ahead, and why the mask is no longer proof Lesson
Causal masking is the mechanism that stops a language model from seeing tokens it is supposed to predict, and prefix invariance is the property it is meant to guarantee. In modern hybrid architectures the mask no longer covers every place information can leak.
An audit finds two released models silently reading future tokens, and the bug makes their own scores look better News
Researchers found that inspecting the attention mask missed all 192 injected causality faults in their tests while a two-forward-pass audit caught every one, and the same audit found real defects in the shipped Zamba2 and Nemotron-H models.
State Space Models and Mamba Lesson
State space models are a transformer alternative that processes a sequence by carrying a fixed-size running summary forward one step at a time, giving them linear cost with length and constant memory per token -- which is why models like Mamba can handle very long inputs cheaply.