News · 2026-07-28

Microsoft lets the video codec pick which pixels the model sees

Microsoft has released Mage-VL, a video-understanding model that uses a video file's own compression decisions to choose which parts of each frame to process. Where a conventional system decodes every frame into pixels and samples them evenly, Mage-VL keeps only the patches the codec spent bits encoding. Microsoft reports over 75% fewer visual tokens and up to a 3.5x wall-clock inference speedup over uniform frame sampling.

Key facts

• Mage-VL is described by Microsoft as a codec-native, proactive-streaming multimodal foundation model with a visual encoder trained from scratch at 4B scale.
• It reports over 75% fewer visual tokens and up to a 3.5x wall-clock speedup versus uniform frame sampling.
• Retained tokens use shared 3D rotary position encoding and feed a Qwen3-4B-Instruct-2507 decoder.
• Primary sources: the model card and the Mage repository.

The problem Mage-VL solves is one of the most wasteful things in multimodal AI. A minute of video contains a lot of frames, and most of what is in them is identical to the frame before. A model that samples frames evenly and then splits each into a grid of patches produces an enormous number of visual tokens, the overwhelming majority describing a wall that has not moved. Since attention cost scales badly with sequence length, that waste is expensive - it is why long-video understanding has been so much harder than long-text understanding. See context windows.

The insight behind Mage-VL is that this problem was solved decades ago by people who were not thinking about AI at all.

Every video codec is, fundamentally, a machine for deciding where the information is. It stores a small number of complete frames - anchor or I-frames - and then, for the frames in between, stores only what changed. It computes motion vectors describing how blocks of pixels moved, and residuals describing what motion could not explain. When a codec allocates lots of bits to a region, it is telling you that something genuinely happened there. When it allocates almost none, it is telling you the region is a repeat.

That bit allocation is a free saliency map, and every existing video model was throwing it away by decoding to pixels first.

Mage-VL reads it instead. All patches from anchor frames are kept - those are the reference points. For predicted frames, only patches where the codec spent bits are retained. The same interface works across H.264/AVC and HEVC/H.265 via motion vectors plus residual energy, and across DCVC-RT via a learned rate map. The surviving tokens are tagged with shared 3D rotary position encoding, so the model knows where in space and when in time each one came from, and then fed to a Qwen3-4B-Instruct-2507 causal decoder. See positional encoding.

The analogy that fits: imagine a courtroom stenographer who transcribes every word of an eight-hour hearing, including four hours of procedural silence, and then hands the transcript to a lawyer to read. Mage-VL is the version where the stenographer's own notes about which moments mattered are handed over alongside - and the lawyer reads only those. The judgment about what was worth recording was already made, for free, by someone whose whole job was making it.

There is a second component that matters for real deployment. A separate lightweight gate handles proactive streaming - deciding when to trigger the full model at all. For a system watching a continuous feed, most of the time the correct action is nothing, and running a 4-billion-parameter encoder to conclude that nothing happened is the same waste at a larger scale.

Two caveats belong on this. The performance numbers are Microsoft's own evaluation, run in part on an 8xB200 setup described in the repository, and no independent group has reproduced them. The repository's citation block calls the work an arXiv preprint but exposes no resolvable arXiv identifier, so the model card and repo are the citable primary sources.

The more interesting limitation is structural, and it follows directly from the mechanism. The codec allocates bits to what changes, which is a good proxy for what matters but not the same thing. A static region can be semantically critical - a fixed caption, an unmoving instrument reading, a price on a screen, a document held steady in frame. Those regions receive few bits precisely because they are stable, and a system that trusts bit allocation as saliency will spend the fewest tokens on them. Anyone deploying this on surveillance, document, or dashboard video should test that case specifically.

Even with those caveats, this is the most portable idea to come out of the day. It requires no new architecture, no new training objective, and no new hardware - only the recognition that compressed video arrives with a saliency signal already attached, and that throwing it away to reconstruct one from scratch is a choice rather than a necessity. Any pipeline currently ingesting compressed video is leaving that on the table.

See also: vision-language-action models and tokenization.

Primary source, verified: read the paper →

Ground Truth. (2026, July 28). Microsoft lets the video codec pick which pixels the model sees. Ground Truth. https://groundtruth.day/news/microsoft-lets-the-video-codec-pick-which-pixels-the-model-sees.html

@misc{groundtruth:microsoft-lets-the-video-codec-pick-which-pixels-the-model-sees,
  title  = {Microsoft lets the video codec pick which pixels the model sees},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/microsoft-lets-the-video-codec-pick-which-pixels-the-model-sees.html}
}

Topics: multimodal · video · efficiency · microsoft · vision-language-models · open-weights