News · 2026-07-13

A benchmark audit finds most video-understanding tests can be aced without watching the video

Video-Oasis, a paper attributed in the day's research to NAVER (arXiv:2603.29616), delivers an uncomfortable audit of the field's video-understanding benchmarks: about 55% of their samples can be answered correctly with no visual input at all. Models are not watching the video - they are exploiting the text of the question, and once those shortcuts are removed, state-of-the-art systems barely outperform random guessing.

Key facts

• Video-Oasis (arXiv:2603.29616, attributed to NAVER) audited existing video-understanding benchmarks.
• Roughly 55% of benchmark samples are solvable without any visual input or temporal context.
• With shortcuts removed, state-of-the-art models perform only marginally above chance.
• The finding implies much reported 'video understanding' is linguistic guessing, not real spatiotemporal reasoning.

The idea of a benchmark shortcut is central here. A benchmark is supposed to measure a specific capability - in this case, whether a model actually understands what happens in a video. But if the questions are written such that the answer can be guessed from the question's wording plus general world knowledge, the benchmark measures something else entirely: the model's grasp of common tropes. Video-Oasis's diagnostic method is simple and damning: strip the video away, give the model only the text, and see how well it does. If it does nearly as well blind as it does watching, the benchmark was never testing vision. For background on why designing honest tests is so hard, see our lesson on how AI is benchmarked.

A concrete example makes the failure vivid. Ask a model 'Is the person in the video jumping?' about a clip from a category where jumping is common, and the model can answer 'yes' correctly at a high rate simply because it knows jumping is a frequent action in that kind of video - without ever processing a single frame. It is not seeing motion; it is playing the odds on what usually happens in videos that prompt that question. Multiply that across a benchmark and you get the 55% figure: more than half the test is guessable from text and priors.

The deeper diagnosis the paper supports is that today's 'video-LLMs' are, functionally, 'text-LLMs with a video-shaped hole.' They bolt a video encoder onto a language model, but the language model's powerful text priors do most of the answering, and the video pathway contributes far less than the benchmark scores suggest. When Video-Oasis removes the linguistic shortcuts - forcing the model to actually rely on the visual and temporal content - the SOTA numbers collapse toward chance. The impressive 'video understanding' was largely sophisticated text completion dressed up as perception.

The analogy: imagine a film-studies exam where the questions are so leading that a student who never watched the movie can still ace it by knowing genre conventions - 'in a horror film, does the character who splits from the group get attacked?' You do not need to have seen this film to answer. A test like that measures genre savvy, not comprehension of the specific work. Video-Oasis is saying much of video-AI evaluation is exactly this kind of exam.

Why it matters: this is the necessary adversarial counterweight to a wave of hype about video-LLMs that 'understand' footage. Progress in a field is only as real as the benchmarks measuring it, and if the benchmarks are gameable, reported gains may be artifacts. It also connects, pointedly, to the day's other vision result: GenCeption's argument that to truly understand motion, a model must first learn to generate it. Video-Oasis diagnoses the disease - current models fake understanding through text priors - and GenCeption proposes the cure - build genuine world knowledge through generation. The caveat: Video-Oasis is a fresh arXiv result, verified in the dossier against the abstract and GitHub repo but not independently reproduced, and its specific 55% figure depends on the benchmarks it chose to audit. But the failure mode it documents - models solving visual tasks without looking - is a well-known and recurring problem in multimodal evaluation, which is why the finding landed.

Primary source, verified: read the paper → (arXiv 2603.29616)

Ground Truth. (2026, July 13). A benchmark audit finds most video-understanding tests can be aced without watching the video. Ground Truth. https://groundtruth.day/news/video-oasis-most-video-benchmarks-need-no-video.html

@misc{groundtruth:video-oasis-most-video-benchmarks-need-no-video,
  title  = {A benchmark audit finds most video-understanding tests can be aced without watching the video},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/video-oasis-most-video-benchmarks-need-no-video.html}
}

Topics: benchmarks · video-understanding · evaluation · multimodal · research