News · 2026-07-20

VideoChat3 Halves Video-Model Latency by Compressing Space and Time First

VideoChat3, a new 4-billion-parameter open video model from Nanjing University's MCG group, roughly halves the time it takes to process a long video by compressing the footage across both space and time before the language model ever sees it. The result is a model that reads far fewer visual tokens than comparable systems while holding its own on a wide range of video-understanding tasks. It is a concrete answer to one of the central costs of video AI: the language model chokes on the sheer number of tokens a video produces.

Key facts

• The gain: on a 2,048-frame input, reported latency drops from 44.4 seconds for Qwen3-VL to 20.4 seconds for VideoChat3 on an H200, per the paper.
• How: 16x spatiotemporal compression yields about half as many visual tokens.
• What shipped: Apache-2.0 weights and three training datasets.
• The catch: the official README still lists training code as unreleased.

The background: a language model that understands video first has to turn frames into tokens, the small chunks it actually processes. Video produces an enormous number of them, and because a transformer's cost grows quadratically with sequence length, more visual tokens means sharply more compute and latency, a pressure covered in the context windows explainer. Most models encode sampled frames independently, which wastes effort on the huge redundancy between neighboring frames.

VideoChat3's trick, an encoder the authors call I3D-ViT, is to apply joint space-time attention within chunks of consecutive frames and pool them along the time axis before handing tokens to the language model. Combined with a pixel-shuffling step, the paper describes 16x spatiotemporal compression, and in a controlled comparison with Qwen3-VL it produces half as many visual tokens. The authors' framing is that they move work out of the language model's expensive quadratic stage and into the cheaper vision encoder. As the paper puts it, the design lets the model "reduce the quadratic sequence-length cost" by front-loading compression. On an NVIDIA H200 using the authors' setup, that shows up as 2,048-frame latency falling from 44.449 seconds for Qwen3-VL to 20.412 seconds.

There is a second idea for live video. VideoChat3 uses three streaming states, silence, standby, and response: in silence it keeps monitoring cheaply, in standby it spends a larger visual budget on the next window because something might be happening, and in response it answers. The verified detail is that standby controls the next window's visual budget, so the model spends compute only when evidence appears. On the authors' streaming ablation, this dynamic policy nearly matched an always-high-budget setting while using a fraction of the visual budget.

Why it matters: efficient long-video understanding is a bottleneck for everything from assistants that watch a screen to models that reason over hours of footage, and VideoChat3 shows a clean architectural lever rather than just a bigger model. On its own tables it is the best listed fully open model on motion and temporal-comparison tasks and improves on 18 of 19 directly comparable offline metrics against Qwen3-VL-4B. But it is not a clean sweep: Molmo2-4B leads some tests, and another model beats it across a proactive-question benchmark. The defensible claim is breadth across motion, long video, grounding, and streaming, not universal leadership, and all these numbers are author-reported, not independently replicated.

The honest caveat is about the "fully open" label. The downloadable model, the standalone I3D-ViT encoder, and all three datasets are tagged Apache-2.0, but the official README marks training code as still unreleased, which conflicts with the paper's present-tense claim that it was released. "Complete datasets" also needs precision: the released Academic2M and other sets provide annotations and mappings, not duplicated source videos, so reproducing the full training mixture still depends on external video access. The architecture and the efficiency idea are the real, inspectable contribution; the reproducibility story has an asterisk until the training code lands.

Primary source, verified: read the paper → (arXiv 2607.14935)

Ground Truth. (2026, July 20). VideoChat3 Halves Video-Model Latency by Compressing Space and Time First. Ground Truth. https://groundtruth.day/news/videochat3-compresses-video-before-the-model-reads-it.html

@misc{groundtruth:videochat3-compresses-video-before-the-model-reads-it,
  title  = {VideoChat3 Halves Video-Model Latency by Compressing Space and Time First},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/videochat3-compresses-video-before-the-model-reads-it.html}
}

Topics: video-models · multimodal · efficiency · open-weights · vision-language