News · 2026-11-16
Gamer research agent controls games through screenshots and keyboard input
miHoYo researchers report that Gamer, a visual game-playing agent, can control games using screenshots and keyboard and mouse actions. In a roughly 50-hour evaluation in Honkai: Star Rail, the authors report that it operated autonomously for more than 95% of evaluation time, although their public account does not fully define that measure.
Key facts
- The Gamer paper was submitted on November 13, 2026.
- The policy observes a screen at 5 frames per second and emits a nominal 30-action-per-second control stream.
- The long-run autonomy percentage measures reported time, not completed quests.
- Primary source: the Gamer paper.
A game agent needs to translate visual scenes and instructions into actions while the world continues to move. Gamer uses a shared interface: it reads screenshots and task instructions, maintains a compact memory, and emits sequences of keyboard and mouse actions. Its policy does not need a privileged game-state feed to choose actions, according to the authors.
The team describes three stages in the paper: training on gameplay across games, instruction tuning with sparse reasoning and memory, then reinforcement learning using outcomes that the environment can verify. Like a player keeping a notebook, the memory records completed objectives, failures and remaining plans before old visual history falls out of the active context.
The game infrastructure still matters. For Honkai experiments, the surrounding system uses engine instrumentation to initialize comparable situations and check outcomes. That is not proof that every action is scripted, but it means the policy's ordinary screen-and-controls interface should not be confused with a wholly game-independent testing environment.
On two held-out side quests within a game represented in training, the reinforced policy scored 80.0% and 83.6% on verified milestones. Those are milestone rates, not the probability of completing an entire quest in one uninterrupted run. The authors also say the longer main-story evaluation was autonomous for “over 95%” of its time, but do not supply a complete denominator or intervention count in that passage.
The project page offers demonstrations and the repository describes the action format. Yet the authors say the Honkai data and trained checkpoints are not public, and the streaming implementation is marked “Coming soon”. The most striking result therefore cannot currently be reproduced from the listed public materials.
The authors also evaluate other games, but transfer depends on how much the setting resembles training. The project's and paper's cross-game aggregate figures differ, and distant unseen games show limited gains. This is evidence of a promising control approach, not a demonstration of unrestricted generality. The distinction matters for any AI agent: what it sees, what it can do and how the environment scores it all affect the result.
A closer look at the evaluation
The authors trained a general control format before specializing on tasks. Their listed continued-pretraining data include roughly 2,400 hours of gameplay, with general multimodal material alongside it; an additional checkpoint includes Honkai recordings. The project also describes a larger source pool. A pool from which data could be drawn is not necessarily the same quantity as the subset used in a particular training run, so those two numbers should not be exchanged.
The held-out side quests contain 30 and 55 milestones, including navigation, menus, dialogue and other actions. Game-native checks determine whether a milestone has been passed. A high fraction of passed milestones is meaningful, but it can hide failures at earlier or later stages and does not show that a single attempt completed the entire quest without a reset. The paper also cautions that a scripted transition may resume a run and that missing evaluations are not counted as failures.
The most generalizable engineering idea may be the division between a compact history and immediate control. A player cannot keep every frame of a long game in working memory; a concise record of location, objective and prior failures may preserve useful information when older observations disappear. But a summary can omit a critical detail or preserve a mistake. The size, reliability and updating of that memory deserve independent tests across settings beyond the game used for the strongest demonstration.
Key questions
Did Gamer finish 95% of quests autonomously?
Can the long-horizon Honkai agent be reproduced from released weights?
Does Gamer generalize to every unseen game?
Cite this
APA
Ground Truth. (2026, November 16). Gamer research agent controls games through screenshots and keyboard input. Ground Truth. https://groundtruth.day/news/gamer-real-time-game-agent.html
BibTeX
@misc{groundtruth:gamer-real-time-game-agent,
title = {Gamer research agent controls games through screenshots and keyboard input},
author = {{Ground Truth}},
year = {2026},
month = {nov},
url = {https://groundtruth.day/news/gamer-real-time-game-agent.html}
}
Comments are replies to this story on Bluesky — reply with any Bluesky account to join in.