News · 2026-08-25

An audit finds two released models silently reading future tokens, and the bug makes their own scores look better

A new audit paper reports that inspecting a model's attention mask - the field's standard check that a language model is not reading ahead - detected zero of 192 deliberately injected causality faults, while a lightweight two-forward-pass audit localized all 192 to the exact layer. Running the same audit against released models turned up real defects in two of them: Zamba2 and Nemotron-H both leak information from future tokens once the input passes a specific length. The failure does not crash anything. It quietly improves the model's own quality metrics.

Key facts

• 192 out of 192 injected faults localized by the new audit; 0 of 192 caught by attention-mask inspection, across eight checkpoints.
• Real defects found in two shipped models: Zamba2-1.2B leaks from sequence length 256, Nemotron-H-8B from 128 - each model's declared chunk size.
• The audit is two forward passes with no training and no gradients, and runs on a CPU in seconds.
• Primary source: The Mask Is Not the Model (arXiv:2608.22876), Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang and Minseo Kim, published August 24, 2026.

The rule the paper formalizes is called prefix invariance, and it is the thing everyone assumes is true: "representations at position t must not depend on future inputs." That is what makes a language model a next-token predictor rather than a very expensive lookup table. If position 40 can see position 41, the model is not predicting, it is copying.

For years, verifying this was easy. Attention was the only operation that mixed information across positions, and attention has an explicit mask - a triangular pattern of allowed and forbidden connections. Look at the mask, confirm it is lower-triangular, done.

That check has quietly stopped covering the model. Modern hybrid architectures interleave attention layers with state-space scans, which mix information across positions by running a recurrence rather than by comparing every position to every other. A scan has no mask to inspect. As the authors write, "attention-mask inspection is incomplete: leaks can occur via scans or normalization despite correct masks." You can hold up a perfectly correct mask while the layer beneath it hands the future to the past. See state space models for how these scans work.

Their replacement check is deliberately unimpressive. Take two inputs that are identical everywhere except the very last position. Run both through the model with hooks on every layer. Report the first layer where the earlier positions diverge past a threshold. If changing the last token changes anything about position five, position five saw the future. No training, no gradients, seconds on a CPU.

The result that makes this newsworthy is not the injected-fault score. It is what happened when they pointed the audit at models people are actually running. A static census of the transformers 5.7.0 source code predicted which released models should leak, and the dynamic audit confirmed the prediction. The defect is a single-axis error: the reference Mamba2 implementation reduces the inter-chunk recurrence over the input chunk axis, while the Zamba2 and Nemotron-H modeling files reduce over the output chunk axis. One wrong axis, and information flows backwards in time.

Ground Truth checked the two named models' shipped configuration files directly. Zamba2-1.2B declares "chunk_size": 256; Nemotron-H-8B-Base-8K declares "chunk_size": 128. Those are exactly the sequence lengths at which the paper says each model begins leaking. The claim matches the artifacts. Both are open weights and easy to inspect yourself: the Zamba2 checkpoint is a single 4.86 GB safetensors file, loaded in bfloat16 per its own model card, which puts a floor of roughly 4.9 GB of resident weights before activations and cache; Nemotron-H-8B ships 16.2 GB of safetensors shards, also bfloat16, so a comparable floor of about 16 GB of weights alone. Neither repository publishes an explicit minimum GPU memory requirement.

Here is why this matters more than a normal bug. A model that can see one token ahead predicts that token better. Better prediction means lower training loss and lower perplexity - which are the exact numbers used to decide whether a training run is working, whether a checkpoint is worth releasing, and how a model ranks. The defect improves the metric that would have caught it. It is a smoke detector wired to switch itself off when there is smoke. The authors argue the obvious conclusion: a causal-correctness certificate belongs next to the parameter count in every model release.

The authors are unusually good about their own limits. The defect lives in the PyTorch chunked-scan fallback path, which runs only when optional fused kernels are absent - so a user with the fused kernels installed may never hit it. Some checkpoints could not be loaded and no claim is made about those. And their audit has two ways of lying to you, both documented: a CLEAN verdict means nothing without a positive control, because some checkpoints return bit-identical outputs for different inputs, and the audit length must exceed the model's chunk size. At their default length of 48, Zamba2 looks perfectly clean, because the buggy code path is never entered.

They also declined to release code, on purpose. The method is about five lines on top of standard forward hooks, and they argue that independent reimplementation is a stronger reproduction than running someone else's binary. Instead they publish complete audit logs, checkpoint identifiers, and per-layer delta arrays. That will irritate people who want a one-command reproduction, and it is a defensible position.

The uncomfortable caveat is scope: this is one team, one audit, one threshold choice, and "diverges beyond a threshold" is a judgment call that determines the entire result. Nobody has independently re-run it. But the specific, named, checkable part - two shipped models whose declared chunk sizes match their reported leak thresholds - holds up, and it took five lines of code to find. Related: one in seven SWE-bench Verified tasks is graded against a patch that does not match.

Primary source, verified: read the paper → (arXiv 2608.22876)

Ground Truth. (2026, August 25). An audit finds two released models silently reading future tokens, and the bug makes their own scores look better. Ground Truth. https://groundtruth.day/news/two-shipped-models-are-reading-tokens-they-should-not-be-able-to-see.html

@misc{groundtruth:two-shipped-models-are-reading-tokens-they-should-not-be-able-to-see,
  title  = {An audit finds two released models silently reading future tokens, and the bug makes their own scores look better},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/two-shipped-models-are-reading-tokens-they-should-not-be-able-to-see.html}
}

Topics: cybersecurity · ai-security · supply-chain · model-auditing · state-space-models · transformers · evaluation