News · 2026-08-08

A video model counted events correctly two-tenths of one percent of the time

A video-language model asked to count simple events in a short clip -- how many times a ball struck a wall, how many times a light blinked -- produced the correct final count 0.2 percent of the time in the hardest condition tested, and recovered only 18.1 percent of the events that actually happened. The model was Google's Gemini 3.6 Flash, and the clips were synthetic videos built so that every event has an executable ground-truth record that can be checked against the model's answer frame by frame.

Key facts

• 2,190 videos across three controlled tasks: bouncing-ball wall contacts, visual blinks, and categorical state transitions.
• In the high-count, high-frequency regime: 0.2% of final counts correct, 18.1% of true events recovered.
• Raising the frame sampling rate lifted bouncing-ball accuracy from 19.6% to 29.3%, but the reported event sequence agreed with ground truth only 3.7% of the time.
• Primary source: The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping.

The design choice that makes this study bite is the refusal to use real video. Real benchmark clips tangle everything together: how many events occur, how fast, how long each lasts, how visually cluttered the scene is. When a model fails, you cannot say which variable broke it. The authors instead render synthetic clips where they can turn two dials independently -- the number of events and their frequency -- while holding the visuals fixed, and where the renderer emits a machine-readable list of exactly what happened and when.

That ledger is the second thing that makes this different. Most video benchmarks grade only the final answer. Here the model's reported event sequence is checked against the true one, timestamp by timestamp. It is the difference between marking a maths exam on the final number and reading the working.

The results come out staged rather than uniform. At an 80 percent reliability threshold, Gemini 3.6 Flash counts persistent state transitions -- something changing colour and staying changed -- reliably up to 12 events at half a hertz and one hertz. For transient blinks, brief flashes that come and go, there is no region of the parameter space where it is reliable at all. The nature of the event determines whether the model can register the evidence in the first place, and that failure compounds as events get more numerous and faster.

Here is the analogy that makes the result intuitive: watching someone deal cards and being asked afterwards how many were red. If you were counting as they went, you know. If you were watching and hoping to reconstruct it later, you are guessing from an impression of redness. The model is doing the second thing. It has no running tally, only a compressed impression of the clip, and a count is exactly the kind of question an impression cannot answer.

The most useful finding for anyone building on these systems is what happened when the researchers threw more frames at the problem. Bouncing-ball accuracy rose from 19.6 to 29.3 percent -- a real improvement on the scoreboard. The reported sequence of events matched the truth 3.7 percent of the time. The model got better at guessing a plausible total without getting meaningfully better at knowing what happened. As the authors put it, extra frames "can therefore inflate final scores without producing faithful event recovery." Anyone evaluating a video system on final-answer accuracy alone is measuring something that can be improved without the underlying capability improving at all.

A same-day paper reports the same shape of failure on the spatial axis. GST-Bench, built from 6,790 minutes of synthetic video with human-verified questions, asks whether a model that has watched a walkthrough of a space knows where things are relative to each other -- self-localisation, object localisation, and scene structure, including mapping what the camera saw onto a top-down view. Across 22 state-of-the-art vision-language models, the best zero-shot score was 42.68 against a human score of 79.08. The authors then built a local variant of the same tasks where the answer is visible in the current view, and models did well on it. Local perception is fine. Consolidating a long stream of observations into one consistent picture is what breaks. Example tasks are illustrated on the project page.

Both results point at the same missing component: persistent state. These models process a video as a large pile of frames inside a context window, not as a stream against which a running record is maintained. A tally, or a map, is precisely the sort of small durable structure that a purely feed-forward pass over frames does not build. It is closely related to what makes world models hard, and to the finding that many video benchmarks can be passed without watching the video at all -- if a benchmark can be solved from priors, it was never testing temporal bookkeeping.

The caveat worth stating: these are synthetic clips of bouncing balls and blinking lights, chosen for controllability, and one could argue they are unrepresentative of real footage. The authors checked, and report that real-world video evaluations show the same concentration of success at low event counts. The counter-argument also cuts the wrong way for the models: if a system cannot count blinks in a clean rendered scene, the failure on a security camera or a surgical recording is unlikely to be smaller. The practical instruction is narrow and immediate -- do not deploy a video model as a counter, an auditor, or a monitor of how many times something occurred, however fluent its description of the footage sounds.

Primary source, verified: read the paper → (arXiv 2608.06361)

Ground Truth. (2026, August 8). A video model counted events correctly two-tenths of one percent of the time. Ground Truth. https://groundtruth.day/news/a-video-model-counted-events-correctly-two-tenths-of-one-percent-of-the-time.html

@misc{groundtruth:a-video-model-counted-events-correctly-two-tenths-of-one-percent-of-the-time,
  title  = {A video model counted events correctly two-tenths of one percent of the time},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/a-video-model-counted-events-correctly-two-tenths-of-one-percent-of-the-time.html}
}

Topics: video · benchmarks · vision-language-models · research · evaluation