News · 2026-08-14

A closed-loop benchmark caught nine world models forgetting the room

A benchmark released this week put nine leading world models under closed-loop evaluation, with an AI agent choosing its own actions toward long-horizon goals rather than replaying a fixed script, and found that every one of them remains unreliable. The specific failures are spatial consistency and persistent state: the models lose track of the geometry of the scene and forget what happened once it moved out of frame.

Key facts

• The anchor number: 171 scenarios, nine world models evaluated, none reliable on long-horizon interactive objectives.
• The method: an "Agent Player" pursues goals in closed loop, so systems are compared on the same objective rather than the same scripted controls. The agent used in the reported experiments is Claude Haiku 4.5.
• When: posted August 13, 2026.
• Primary source: PlayWorld, arXiv:2608.13552.

A world model is a system that predicts what happens next in a scene. Show it a frame and an action, and it produces the frame that should follow. The ambition behind them is enormous: if a robot can accurately imagine the consequence of an action before taking it, it can plan without breaking anything, and it can learn from imagined experience rather than expensive real trials.

Evaluating them has been the weak point. The standard approach replays a fixed sequence of actions through the model and scores how good the resulting video looks. That measures whether the output is plausible, which is not the same as measuring whether it is correct. A model that produces gorgeous, physically incoherent video scores well.

PlayWorld changes the question. Instead of a script, it puts an agent in the loop that pursues a long-horizon objective, choosing its own actions as it goes. Systems are then compared on whether the goal was achieved, not on whether they followed the same controls. The scoring dimensions are named precisely: geometry consistency, interaction fidelity, "out-of-sight evolution," insight evolution, plus conventional video quality and controllability measures.

That third one, out-of-sight evolution, is the interesting one and the one models fail. It asks whether the world keeps behaving sensibly in the parts you are not looking at. Turn the camera away from a table and turn back; is the cup still there, in the same place, at the same angle? Humans take this for granted. A video generator, which is fundamentally producing plausible pixels frame by frame, has no particular reason to enforce it, and generally does not.

Across 171 scenarios and nine systems, including Genie 3, LingBot-World, HY-World2, SANA-WM, Hunyuan-GameCraft-2 and Matrix-Game-3.0, the finding holds. Long-horizon interactive objectives remain out of reach, with spatial consistency and persistent state the specific weaknesses.

A companion benchmark released the same week attacks a different gap. H2R-Bench asks whether video of a human doing a manipulation task can be transformed into video of a specific robot doing the same task, which matters because human demonstration video is abundant and robot demonstration video is not. The setup is careful: 120 source videos expanded into 240 transfer cases by pairing each with two target robot bodies, spanning six families of manipulation, with 11 video generators evaluated and 660 generated videos checked by humans, reaching a macro-average correlation of 0.883 between human and automated judgments. The result is again negative. Current systems struggle with embodiment correctness, with functional contact, and with actually completing the task.

Two positive results anchor the cluster. AtlasVLA is the one that reaches real robots. Instead of reacting frame by frame, it maintains two separate memories: a persistent four-dimensional world state holding workspace geometry, and a working memory tracking task progress. Wrist-camera observations are lifted into a voxel-hashed spatial representation, and a diffusion transformer is conditioned on both memories together. It reports results on standard robot benchmarks and on real long-horizon manipulation tasks using only a wrist-mounted camera, and its ablations confirm the point: remove either memory and success drops. That is the clearest evidence in the set that the diagnosis, missing persistent state, matches the cure.

DreamX-Phi 1.0 attacks faithfulness from the other end. Its rollouts are conditioned on a prescribed two-armed action trajectory, with geometric encoding that injects each arm's rigid-body transform directly into the model's attention, so the predicted video follows the commanded motion rather than drifting toward whatever the language prompt suggests. A depth branch and additional supervision keep object identity stable through grasping. It ranks first on one track of a world-model arena and second on another. One correction worth carrying: despite early summaries, the paper reports no evaluation on physical robot hardware.

The honest caveat is that a cluster of benchmarks reporting failure is a healthy sign, not a crisis. It means the field has stopped grading itself on whether the video looks nice and started grading itself on whether the imagined world holds together, which is the property that actually matters for planning. The convergence here is on an interface and an evaluation style, not on a single architecture, and the five papers address genuinely different subproblems rather than restating one advance.

Ground Truth has followed this thread: world models forget, seven new papers cannot agree what a world model is made of, and background on vision-language-action models.

Primary source, verified: read the paper → (arXiv 2608.13552)

Ground Truth. (2026, August 14). A closed-loop benchmark caught nine world models forgetting the room. Ground Truth. https://groundtruth.day/news/a-closed-loop-benchmark-caught-nine-world-models-forgetting-the-room.html

@misc{groundtruth:a-closed-loop-benchmark-caught-nine-world-models-forgetting-the-room,
  title  = {A closed-loop benchmark caught nine world models forgetting the room},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/a-closed-loop-benchmark-caught-nine-world-models-forgetting-the-room.html}
}

Topics: world-models · robotics · benchmarks · video-generation · evaluation