Ground Truth.
AI, checked against the source.

How AI Learns to Play Games: Watching, Acting and Adapting

2026-11-16 · Breach Protocol: Inside the AI Blackbox — full transcript

Learn how game agents turn screen observations into controls, learn from demonstrations and rewards, and manage long tasks while a game keeps moving. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio

Listen on Spotify · Watch on YouTube · Pocket Casts

The screen-to-action loop

Eris: <|prosody:expressive_high|>A miHoYo team reports that its game-playing agent Gamer passed more verified milestones on two Honkai: Star Rail side quests after feedback training than before it. Gamer learned to act from recorded play, then improved with feedback as it played: it sees the screen and uses keyboard and mouse controls. Those milestones are pieces of quests, so that comparison doesn't tell us <|prosody:pause|>how often the agent completed a whole quest without interruption.

Vestra: An agent is a system that receives information, takes actions, and encounters the consequences. Here the environment is the game and everything in it that responds to the controls. An observation is the information supplied to the agent, such as a screenshot. An action is a control it sends, such as holding a movement key, clicking, or moving the mouse. The task is what it's trying to accomplish: reach a character, for example, and start a conversation.

Eris: The loop starts with an image of the game. The agent uses that image, its instructions and whatever history it has available to decide what the situation calls for. It might move the mouse toward a doorway, hold forward briefly, then receive another image and see whether it got closer. Its policy is the learned rule for choosing controls in a situation. It can depend on the current screen, the goal, and earlier observations and actions. <|prosody:expressive_high|>But how much can that picture actually tell it?

Vestra: The game internally knows things the picture may not show: what's behind a wall, whether an unseen counter has changed, or where a moving object was a moment ago. Two different game states can produce <|prosody:pause|>similar screens. A sequence of observations can help reveal movement, but a screen-based agent still has to work with partial information. It can't simply read everything the game knows.

Eris: <|prosody:expressive_high|>That's what makes the doorway example less simple than it sounds. And the time between looking and acting matters. If the agent holds forward too long, it may walk past the door. If the game keeps running while the model calculates its next control, an enemy could move before that control arrives. The observation was accurate when captured, but the action can already be late.

Learning controls from demonstrations

Eris: One way to build a policy is imitation learning: show it examples of people playing, paired with the controls they used, and train it to predict those controls in similar situations. If many examples show a player looking at a tree and holding the attack button, the policy can learn that association. It doesn't need to discover every useful motion by randomly pressing keys.

Vestra: But ordinary gameplay video rarely records which buttons were pressed. In the Video PreTraining study, OpenAI researchers tackled that missing-label problem in Minecraft. They collected a smaller set of human play with both the screen recording and the actual keyboard and mouse inputs. <|emotion:enthusiasm|>From that they trained an inverse dynamics model: a separate model that looks at video before and after a change and predicts which control probably caused it.

Eris: So if the view turns to the right, the labeler can look at both sides of the turn and infer a likely mouse movement. The agent that eventually plays doesn't get to see the future screen; it has to choose using the present and past. <|emotion:surprise|>The labeler can use the future because it's working on a recording that already exists.

Vestra: <|emotion:enthusiasm|>That's the part I like: using the recording's later frames to estimate the missing controls. The researchers applied that labeler to a much larger collection of online Minecraft videos and trained a playing policy on the resulting screen-and-predicted-control pairs. <|prosody:expressive_high|>Those inferred controls are estimates, not recovered ground truth. The authors report that the resulting policy could collect logs and sometimes make a crafting table in fresh survival games. Compared with people in their study, it made crafting tables much less often. Further imitation training on examples focused on early play helped it make wooden and stone tools.

Eris: <|prosody:expressive_high|>There's a useful distinction there: watching taught the policy how human play tends to unfold, not necessarily how to obey any new request. The main Minecraft policy wasn't instructed to perform arbitrary tasks. And if it makes a small mistake, it may see a situation that barely appeared in the demonstrations. Another uncertain action can push it further away, so errors accumulate. Rare hazards and unusual controls can be missing from the examples altogether. Could text instructions give it more direction?

Vestra: The researchers also experimented with a version conditioned on text. Some prompts encouraged exploration or collecting common materials, but prompts to hunt or gather flowers didn't work reliably, and asking for a specific crafted item had <|prosody:pause|>no significant effect in their test. The authors described that steering as too weak for practical use.

Learning from rewards

Vestra: Reinforcement learning changes the source of instruction. Instead of simply predicting a demonstrator's next control, the agent tries actions in an environment and receives a reward signal: a number indicating how the outcome relates to the objective. Training adjusts the policy so actions associated with better outcomes become more likely. A reward is a measurement chosen for training, not a feeling, and it doesn't have to describe every detail of what happened.

Eris: In a DeepMind study of seven classic Atari video games, an agent learned from screen images and changes in score. Its network estimated how much future reward each available action might lead to, then used those estimates to choose a control. The researchers stored past experiences and sampled them again during training, so the model could learn from more than just the latest consecutive moments. They used the same basic architecture and learning method across the games, trained on each game, and reported stronger performance than earlier learning approaches on most of the games they tested. It was still far from the human comparison on several.

Vestra: A delayed reward makes learning difficult. Imagine the only feedback arrives when a player completes a long quest. If the agent never gets there, almost every attempt produces the same uninformative result. <|prosody:expressive_high|>Even when one succeeds, which of the hundreds of earlier movements made the difference? Exploration means trying actions that might lead to new outcomes; if the useful sequence is extremely unlikely, exploration may never reach the reward.

Eris: <|emotion:enthusiasm|>That makes me appreciate the demonstrations more. Demonstrations can give that search a head start. In the Minecraft study, reinforcement learning from a random policy made almost no progress toward a diamond pickaxe. Starting with the imitation-trained policy, and focusing it on early-game examples before reward training, did better. The researchers rewarded intermediate items along the crafting sequence, and the agent sometimes made the diamond pickaxe, though success remained rare. <|prosody:expressive_high|>That comparison is about this setup, not a promise that reinforcement learning always improves imitation.

Vestra: Even intermediate rewards can mislead. The paper describes how rushing to collect stone while leaving a crafting table behind may earn an early reward but make the next tool harder to craft.

Eris: <|sfx:sigh|>Uh, that stone reward bothers me: it looks like progress while making the next step harder. What happens to the skills the agent already learned?

Vestra: A reward signal is only as useful as the behavior it encourages, and further training can also erode skills learned earlier. The Minecraft researchers added a constraint to help preserve those skills.

From controls to longer goals

Eris: A key press is tiny compared with an instruction like find a character, speak to them, and return with an item. The agent has to connect the goal to many observations and controls, and notice when the next part of the task has changed.

Vestra: Gamer's authors trained that connection in stages. First, they adapted a visual and language model using gameplay recordings with aligned controls from multiple settings. <|emotion:enthusiasm|>They included a description of what each game's controls do, and varied key mappings in training to encourage learning the function of a control instead of just its usual letter. <|prosody:expressive_high|>A different game or a changed binding can still confuse it.

Eris: Then they trained it on instructions at different scales: a local operation such as choosing a menu option, a subgoal such as reaching and interacting with a person, and a longer task made of several subgoals. The authors used another language model to annotate recordings with instructions, selected decision points and summaries, with human examples guiding the annotations. That makes the training material more structured, but annotations can also be wrong or incomplete.

Vestra: The policy can output a short piece of explicit reasoning when it faces a significant decision, and act directly during routine movement. It also maintains a written summary of earlier play. That summary can include completed objectives, failed attempts, useful locations, resources and the remaining plan. As older detailed observations leave the limited recent context the model can process, the summary carries some information forward alongside recent screens.

Eris: <|emotion:amusement|>I like the practical tradeoff: the agent doesn't need to narrate every step down a corridor, but it does need to know why it entered the corridor.

Vestra: <|sfx:humming|>Hmm, but would you trust that summary to remember the right things?

Eris: <|prosody:expressive_high|>That's the part that worries me. A summary is a compressed record. If it leaves out a locked door or mistakenly says an item was collected, the next plan can inherit that mistake. Having a memory mechanism doesn't guarantee correct memory or successful planning.

Learning and acting while the game runs

Vestra: Gamer's final training stage uses online reinforcement learning: the current policy plays attempts in the game, receives verified feedback, and is updated from those attempts. In the authors' comparison, this stage improved the share of verified milestones passed on both held-out side quests relative to the instruction-trained version. The quests were new to the evaluation, but the model had trained on other play in the same game.

Eris: <|prosody:expressive_high|>How can feedback reach across a task that's too long to fit in the model's recent context?

Vestra: The researchers divide attempts into shorter segments connected by the generated memory summaries. They compare outcomes for segments at corresponding stages of separate attempts at the same task, and use the relative results to update the policy. Where the game can verify a milestone, it can provide feedback before the whole quest ends. The method gives later actions within a segment more weight when assigning the outcome, but that's a training assumption, <|prosody:pause|>not proof that those actions caused the result. Different attempts may also reach a stage in different conditions.

Eris: The other constraint is the clock. In a paused test, the world waits while the model works. In real time, a jump or a click may have to arrive before a moving target disappears. Looking and deciding frequently can catch changes sooner, but every decision consumes computation. Longer action sequences reduce how often the model must decide, but they can drift if the world changes before the next observation. How does Gamer handle that tradeoff?

Vestra: Gamer samples a screen, then produces a sequence of six short key and mouse control blocks. Those blocks can represent a held key or a quick tap between observations, and the game keeps advancing while they execute. <|emotion:enthusiasm|>Its serving system reuses parts of the recent computation and starts sending verified controls before the entire action packet has finished generating. The authors report that their faster real-time setup learned more effectively than a slower control setup on selected navigation tasks while the game continued to run. <|prosody:expressive_high|>That's evidence about their tested engineering choices, not a universal solution to latency or long-term planning.

Eris: <|emotion:surprise|>And a missed image can matter as much as a slow response. In the study of classic Atari video games, the researchers changed how often the agent sampled the screen in Space Invaders because a blinking laser could otherwise be invisible at the chosen sampling interval. A policy can't react to a cue that never reaches its observations.

Reading a game-agent claim

Eris: The controls are only one part of the experiment. Gamer's policy receives screen images, task context and control descriptions, and sends keyboard and mouse actions. For its Honkai experiments, the researchers also instrumented the game engine: the surrounding test system sets up comparable starting situations and checks whether objectives were achieved. <|prosody:expressive_high|>The policy's screen-based view and that privileged experimental support are different things.

Vestra: That distinction matters when interpreting the results. The authors' side-quest numbers count verified milestones, <|prosody:pause|>not uninterrupted whole-quest completions. They say the system operated autonomously for most of the time in a much longer main-story evaluation, but that passage doesn't fully specify the time calculation or intervention count.

Eris: <|prosody:expressive_high|>That's frustrating: I want to know where it needed help. How are interruptions and missing evaluations handled?

Vestra: The paper also warns that a run can resume through a scripted transition and that missing evaluations are not counted as failures. Those details limit what we can infer about sustained independent play.

Eris: Their cross-game experiments also show why a shared keyboard and mouse interface isn't the same as universal skill. They report useful transfer to some related settings, but limited gains in more distant unseen games. The strongest Honkai-trained checkpoints and data aren't publicly released, so the most specific results can't be reproduced from those materials alone. <|prosody:expressive_high|>These are the authors' reports, not independent replication.

Vestra: When you hear a game-agent claim, start with what it sees and controls. Then ask what demonstrations or other experience it learned from, what counts as success, and whether the game keeps moving during inference. Check how many complete attempts were evaluated, whether failures and interruptions are included, and what people or game-engine tools set up and verify the test. Finally, look for a genuinely new task or environment, and compare it with what the agent already trained on.

Eris: Ambiguous screens, missing rare examples, a badly chosen reward, exploration that never finds the right action, accumulated mistakes and changed controls can each break a different part of the loop. Game agents are interactive AI systems: they observe, act and receive new information. <|prosody:expressive_high|>A result in a game doesn't by itself establish reliable computer use, robotics or general autonomy. Those claims need their own tests in their own environments.

Sources and follow

Vestra: <|prosody:expressive_high|>For more of today's AI stories, find the separate news brief and visit Ground Truth at groundtruth.day.

Eris: <|emotion:enthusiasm|>What would you want a game agent to learn next? Tell us in the comments, and follow AI in Detail for more explanations of the research. <|emotion:contentment|>Thanks for listening.