News · 2026-07-24

Humans score 96% on a new visual exam. The best model gets one in ten.

A new benchmark called ActiveVision asks models to keep re-examining a picture while they reason through it, and the results are lopsided: GPT-5.5 solved 9 of 85 problems at its highest exposed reasoning setting, while unaided humans averaged 81.7 out of 85. The model scored zero on 11 of the 17 task types. The gap is not about recognising objects - it is about whether a model can keep looking.

Key facts

• The anchor number: 9 of 85 problems solved by the strongest tested model, against a human average of 81.7 of 85.
• When: the paper and benchmark surfaced on July 23-24, 2026.
• Who: the authors of An Exam for Active Observers, with a companion paper from a ZJU-affiliated group on the same weakness.
• Primary source: arXiv:2607.16165 and the official benchmark page.

Almost every visual benchmark asks a version of "what is in this picture?" ActiveVision asks something stranger and, it turns out, much harder. Count every separate region or closed loop in a photorealistic scene. Follow one marked curve through a tangle of crossings and report, in order, the labels you pass. Find every exact silhouette match against distant candidates. Identify which cells differ between two nearly identical fields. Eighty-five problems, five each across 17 task families, each with an exact answer.

The design claim is precise and it is the interesting part. No single pass of looking - no caption, however detailed - can retain enough arbitrary positions, shapes and paths to solve these. The solver has to keep returning to the pixels while it works, the way you trace a wire with your finger rather than memorising the whole diagram and reasoning about it from memory. That is what "active" means here: vision as something you do repeatedly during reasoning, not a single perception step that hands text off to a language model.

Models appear to do the second thing. The failure patterns the authors report all look like reasoning over a fading snapshot rather than over the image: systematic undercounting in crowded scenes, losing the correct path almost immediately when tracing, and defaulting to "no difference" on fine comparison tasks. Cranking up reasoning effort increased spend sharply and barely moved the score - which is itself a meaningful finding, because test-time compute has been the reliable lever almost everywhere else this year. You cannot think your way to a detail you never encoded.

The tool-augmented result is the sharpest part of the paper. Coding agents allowed to threshold the image, crop templates and run connected-components analysis reached roughly half the set - a huge jump - but took around a quarter of an hour per problem against a human's rough half-minute. And on realistic textures, the masks and tracers frequently failed, with the agents often failing to visually notice that their own overlay was wrong. A system that can write correct image-processing code but cannot see that its output is garbage has an inspection problem, not a programming one.

A companion paper, Show, Don't Tell, attacks a related but distinct bottleneck: whether forcing a model to serialise a spatial judgment into coordinates or prose throws away the judgment. Its ProVisE framework lets an image-generating model answer visually instead - marking points, drawing paths, rendering depth, masking regions - then uses a fixed, task-specific parser to convert the picture back into the benchmark's native metric. The result is genuinely nuanced. Answering visually helps when the answer remains directly visible as geometry, like depth or grounded object relations. Text-based vision-language models stay better when the task requires composing constraints, shifting perspective, judging feasibility, or updating state. Drawing preserves a spatial conclusion; it does not manufacture the reasoning needed to reach one.

Put the two together and you get two separate diagnoses rather than one sweeping verdict. Models do not reliably keep consulting dense visual evidence while they reason. And even when a model does hold a spatial judgment, squeezing it through text can discard the useful part. Neither paper shows that AI "cannot see" - these are the same systems that read documents and describe scenes competently. They show that the perception-to-reasoning handoff is one-shot and lossy, and that the losses are catastrophic exactly where the task needs repeated inspection.

The research direction this points at is worth tracking, because it connects to how world models and embodied systems are being built. If a robot's visual reasoning collapses the moment a task requires re-checking the scene mid-plan, that is a much more consequential failure than a low score on a puzzle set.

The honest caveats are real. Both papers are unreviewed preprints. ActiveVision's images are procedurally generated and then re-rendered photorealistically, which makes the ground-truth labels exact but keeps the whole thing a synthetic-to-realistic pipeline rather than sampled reality. The headline comparison is a single benchmark run by the authors, not an independent replication, and the human panel size is not specified in the primary material. And what the benchmark measures is one specific competence - reliably revisiting visual evidence during solving - not all of spatial intelligence. The gap is dramatic and the diagnosis is plausible. It is not yet a settled fact about what these models fundamentally lack.

Primary source, verified: read the paper → (arXiv 2607.16165)

Ground Truth. (2026, July 24). Humans score 96% on a new visual exam. The best model gets one in ten.. Ground Truth. https://groundtruth.day/news/activevision-humans-96-models-10.html

@misc{groundtruth:activevision-humans-96-models-10,
  title  = {Humans score 96% on a new visual exam. The best model gets one in ten.},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/activevision-humans-96-models-10.html}
}

Topics: benchmarks · vision · multimodal · evaluation · reasoning · papers