How AI Makes 3D Characters Move

2026-11-07 · Breach Protocol: Inside the AI Blackbox — full transcript

From hand animation and motion capture to text-driven diffusion, interactive motion and smart primitives: how Kimodo, ARDY, ARDY Mini and MotionBricks generate and control moving characters. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio

Listen on Spotify · Watch on YouTube · Pocket Casts

From a sentence to a moving skeleton

Eris: <|prosody:expressive_high|>NVIDIA's ARDY research system generates 3D character motion in short chunks while responding to changing instructions, such as a new path or a pose to reach. It learns from examples of people moving. To see what the system has to control, imagine asking a character to walk to a chair and sit: it needs a pose for every frame, including where its body goes and how its joints move. What does a pose actually specify?

Vestra: Think of a skeleton as a hierarchy of joints connected by bones: a shoulder connects to an arm, which connects to an elbow, and so on. A pose is the arrangement at one instant. Positions say where joints are; rotations say which way parts are turned. Rotating a shoulder changes where the arm and hand end up. A sequence of poses played at a frame rate, the number of poses shown each second, becomes motion. At twenty frames a second, forty frames represent two seconds.

Eris: There's also the root, a reference for where the whole character is and which way it faces in the scene. If the root travels forward while the legs cycle, the person walks across the floor instead of walking on the spot. And when a foot is supposed to be planted, its location should stay put. Otherwise the foot slides across the ground even though the leg looks as if it is taking a step. <|prosody:expressive_high|>That sliding really bothers me.

Vestra: Traditionally an animator could set important poses, called keyframes, and shape the movement between them. Or a performer could be recorded with motion capture, which tracks a real person's movement for use on a digital skeleton. Both can produce convincing work, but recording needs performers, equipment and cleanup, and hand authoring takes skilled labor. <|prosody:pause|>A recording also gives you the particular action performed, not every possible variation or path a game might need.

Eris: The human-motion survey we are using covers work through 2023. It describes a move from manually designed and example-based methods toward learned models that discover patterns in motion data. Motion graphs connect recorded clips at compatible points; later motion matching searches a library for frames that suit the character's current state and controls. Those are valuable ways to reuse performances, but large libraries and the transitions between actions take work to manage. Can learning from those examples help with the transitions?

Vestra: <|prosody:expressive_high|>Learned models offered another way to generalize from examples. Earlier neural systems predicted motion using the recent past and control signals; the MotionBricks paper's account points to a 2017 network for character navigation. Instead of an artist spelling out every transition, training adjusts a network so that its predictions resemble many examples. <|prosody:pause|>It still inherits the coverage and limitations of those examples, and early interactive networks could be specialized to particular tasks.

Eris: Then descriptions became part of the training signal. Collections paired motion sequences with words describing the action, so a model could learn an association between a sentence and plausible movement. <|emotion:amusement|>That does not make the sentence a complete choreography: “walk happily” leaves many possible steps, speeds and arm movements. A generative motion model can produce different plausible sequences instead of merely selecting one fixed recording.

Vestra: The 2022 Human Motion Diffusion Model paper is one example of a new family. Diffusion trains a model to reverse a gradual corruption process. During training, noise is added to real motion and the network learns to recover clean motion with help from a description. At generation time it starts from random noise and repeatedly cleans its guess into a sequence of poses. Different starting noise can give different motions for the same words.

Eris: <|prosody:expressive_high|>I like that room for variation. A good-looking short clip is only the beginning. What happens when a character also needs to follow a route, reach a specific pose, and keep its footing?

Learning motion and controlling it

Eris: A sentence tells a motion model what kind of action to make. An animator may also need to tell it where that action happens: walk along this path, put a hand here, or be in this pose at a particular moment. Those specifications are constraints. The challenge is to meet them without making the rest of the movement stiff or implausible. <|prosody:expressive_high|>Can the model refine all that while someone is steering?

Vestra: <|prosody:expressive_high|>Diffusion gives a model repeated chances to refine a motion, but each cleaning step costs computation.<|prosody:pause|> Kimodo uses that process for offline authoring: the animator can wait for a whole clip, then inspect or edit it. Its paper says generation can take several seconds. That's a different requirement from a game character that must react while the player is steering.

Eris: Kimodo's researchers trained on a large motion-capture collection with text descriptions and used more examples, a larger model and larger training batches to study scale. The more varied training examples helped reduce foot sliding and improve how accurately the model met constraints. <|prosody:expressive_high|>Some summary measures changed much less, so the researchers caution that a single quality score can miss differences that matter to an animator.

Vestra: <|emotion:enthusiasm|>The route-and-pose split is the part I like here. Kimodo separates the body's overall route from the detailed pose. It predicts a root path and then conditions the body movement on it, repeating those stages during denoising. In its comparison, a similarly sized single-stage model followed text about as well, but slid its feet more and missed constraints by more. The division makes coordinating the feet with the route easier for this model.

Eris: <|prosody:expressive_high|>That separation matters because a person walking a straight route still shifts their hips side to side. <|emotion:surprise|>If you force the actual pelvis to trace a mathematically straight line, the paper says the resulting walk can look like sneaking. <|prosody:expressive_high|>A smoothed guide lets the body sway around the route, though the pelvis can consequently drift away from the specified line.

Vestra: That drift is why I wouldn’t equate a better-looking walk with exact path control. There are other ways to represent motion. A latent representation is a learned compact description of data: rather than handing a model every joint value separately, another network can compress patterns and later decode them back into poses.

Eris: When these papers talk about motion tokens, are those compact descriptions all the same kind of thing?

Vestra: A token, in this setting, is a small unit of that representation. Some systems choose tokens from a discrete set, rather like choosing entries from a vocabulary; other systems use continuous numerical representations.

Eris: The later papers describe models inspired by language generation that predict motion tokens in sequence, and masked models that fill in missing tokens. Autoregressive means using what has already been generated to generate the next piece. Those approaches can help organize a long sequence, but the ARDY authors note that earlier language-model-style motion systems were generally designed for offline generation and did not offer precise spatial control. <|prosody:pause|>A token doesn't automatically solve speed or control.

Making motion respond in real time

Eris: ARDY takes the control problem into an interactive setting. NVIDIA's system generates a short window of motion, plays it, and generates more using the motion already produced. While it runs, someone can change the text or supply a path, a waypoint, or a pose to reach. The aim is to keep the motion flowing while the instruction changes. <|emotion:enthusiasm|>That’s the flexibility I want from the controls.

Vestra: <|prosody:expressive_high|>And the representation matters for that control. ARDY combines two representations. It keeps the root's position and heading explicit, so a route or target can be expressed directly in scene coordinates. It compresses the body's detailed motion into learned latent features, which are cheaper for the generative model to handle. A decoder turns those features back into joint motion. Inside each diffusion step, one stage predicts the root and a second predicts the body using that root; the stages work together across the repeated cleanup steps.

Eris: <|prosody:expressive_high|>What if the target is beyond the next window? If the destination is several seconds away, the character needs to start moving in its direction now.

Vestra: ARDY can include future constraints beyond the chunk it is currently generating. It also takes a variable history of past motion. For “walk, pick something up, then keep walking,” history can help distinguish the first walk from the one after the pickup.

Eris: How far can that look-ahead go? I’d worry about treating it as unlimited foresight.

Vestra: <|sfx:humming|>Hmm, its memory is bounded in the demo: very distant targets aren't considered until they enter the configured future window. Longer history can help complex instructions, but it also takes work to process. The paper reports that very short generation windows hurt quality and control, while a shorter useful window could adapt faster to a changed prompt. <|prosody:expressive_high|>That is a responsiveness trade-off, not unlimited foresight.

Eris: The team tested text matching, motion similarity, foot sliding and distance from specified joint targets. On a public motion dataset, ARDY's long-horizon comparison with an earlier interactive diffusion model favored ARDY, and people comparing examples preferred it on motion quality, text alignment and reaching the target. <|prosody:pause|>Those tests cover selected actions and goals, not every game scene.

Vestra: In the authors' interactive demo, a four-step version ran on an NVIDIA RTX 4090; a ten-step version gave somewhat better control at greater latency. A buffer of already generated frames can keep playing while the next chunk is computed. Neither the latency nor the paper's results promise the same performance on every machine.

Eris: <|prosody:expressive_high|>The demo makes this appealing. How much of that carries over to making a physical body perform the motion?

Vestra: ARDY predicts movement rather than forces, and the authors still observe artifacts such as sliding and jitter. A physical robot or simulated body needs additional control that can actually execute the motion.

Building movement from targets

Eris: MotionBricks takes a different route to interactivity. <|prosody:expressive_high|>It doesn't take a sentence as its main control. It generates movement between recent poses and target poses, a task called in-betweening. A higher-level interface turns a player's steering or an object interaction into those targets. The idea is to combine flexible generation with instructions that a game can specify precisely.

Vestra: The system first predicts timing and an initial root trajectory. A separate pose module predicts compact discrete motion tokens, and a decoder uses those tokens, the trajectory and keyframes to reconstruct the moving body. <|emotion:enthusiasm|>Its tokenizer uses several learned groups of choices rather than one giant list. In the authors' tests, that design handled a growing motion collection better than their single-group comparison and was more robust to some token prediction errors.

Eris: The higher-level controls are what the authors call smart primitives. Smart locomotion takes a direction and speed, along with a reference pose for a style such as crouching. It proposes a route, and the learned modules refine it to fit the body and the style. The controller can replan when the instruction changes or the supply of generated frames runs low. What does the target setup look like for an object interaction?

Vestra: A smart object associates target poses and interaction logic with something in the scene. For a ledge, a key pose might bring the hands to the edge, followed by a pose standing above it. The target poses can be anchored to the ledge, so the same setup can adapt to different approaches. Some targets are strict, while others guide the motion without requiring an exact pose.

Eris: <|emotion:enthusiasm|>I like that anchoring the poses to the ledge lets the setup adapt to different approaches. How much of that interaction still has to be supplied?

Vestra: Someone still has to provide those poses and define the interaction; the system does not infer a complete game from a sentence.

Eris: In tests of filling motion between held-out keyframes, the researchers compared several methods on speed, reaching targets, smoothness, foot contact and similarity to recorded motion. Their system was preferred in their participant study and reached targets reliably on their large dataset. <|prosody:pause|>Its reported very fast inference was measured on an RTX 5090, and is a model measurement under that setup, not the total cost of a finished game.

Vestra: The limitations are concrete. Rare actions were thinly represented in the training collection, and the paper says object interactions were not explicitly modeled as object motion in the data. <|prosody:expressive_high|>The generated skeleton isn't a physics simulation: it can collide with itself or propose a movement a robot cannot safely perform.

Eris: I’d worry about a robot being asked to perform an unsafe movement. What executes the motion in the demonstration?

Vestra: The robot demonstration uses a separate tracking controller. The authors also say real-world object interaction would need perception of the scene rather than privileged knowledge of exact object positions.

Shrinking the text path

Eris: <|prosody:expressive_high|>ARDY Mini asks a narrower question: can we keep ARDY's motion generator but make its text understanding much smaller? It's an unofficial community derivative by GitHub user intsuc, not an NVIDIA project, and it has no paper. <|prosody:pause|>What we know comes from its code and its own documentation, not peer review.

Vestra: Original ARDY uses a large language model, Llama 3 eight billion, through a text encoder called LLM2Vec. The encoder turns a prompt into numerical features for the motion model. Mini replaces that text encoder with a smaller sentence model called MiniLM, trained to imitate the large encoder's useful output. That teacher-and-student training is called distillation. The particular ARDY motion checkpoint remains the same: it produces twenty frames a second in forty-frame windows.

Eris: <|prosody:expressive_high|>The student doesn't simply copy the teacher's original features. ARDY's root and body stages each transform those features differently before using them. Mini learns to predict both transformed results directly, packaging the root and body conditions together. <|emotion:enthusiasm|>That's a clever place to imitate the teacher: it's what the two stages actually need.

Vestra: <|prosody:expressive_high|>That choice makes a subtle detail in the code matter. Each original transformation also adds a fixed learned offset, called a bias. ARDY sometimes compares a text-conditioned prediction with a no-text prediction to strengthen the effect of the prompt. Its no-text input is made by setting the text features to zero, but the original transformation still adds its bias.

Eris: <|emotion:surprise|>Even the no-text case has that offset. How does Mini preserve that comparison?

Vestra: Mini learns the two results before that bias, and the modified projection adds the original bias back in both cases. <|emotion:relief|>That preserves the intended no-text comparison. <|prosody:expressive_high|>Because those transformations belong to a particular checkpoint, the student is checkpoint-specific, and the loader rejects mismatched models.

Eris: The author trained on motion descriptions from NVIDIA's SEED timeline annotations, removed duplicate descriptions and split related recording groups so held-out prompts were separate. The teacher's answers were cached once.<|prosody:pause|> The student's training combined matching the values, matching their directions, and preserving relationships between similar prompts. It adjusted the pretrained sentence model more gently than its new output layers.

Vestra: The author reports that, on one NVIDIA GB10 machine, the text-conditioning path loaded and encoded prompts many times faster and used a small fraction of the memory. <|prosody:pause|>That timing excludes motion generation and rendering. In paired tests using the same prompts and random starting conditions, student-generated motions differed from the teacher's by much less than teacher motions differed across different random seeds; predicted foot contacts also agreed often.

Eris: I’m encouraged by that closeness. Does it tell us whether the motion follows the words?

Vestra: <|prosody:expressive_high|>That's a replacement-fidelity test, not proof that either motion actually obeys the words or a reproduction of NVIDIA's paper benchmark. The author reports the results; they haven't been independently reproduced here.

From generated skeleton to character

Eris: ARDY Mini also has a browser demo. It runs three exported networks: the text encoder, the motion denoiser and the decoder. They run on the user's device through WebGPU, a browser interface for graphics hardware. <|prosody:pause|>There is no CPU fallback for inference. If the graphics adapter supports half-precision operations, the app selects a mixed-precision version; otherwise it uses a full-precision version, still through WebGPU.

Vestra: Half precision stores numbers with fewer bits. It can reduce size, but not every calculation tolerates the reduced range and precision. <|prosody:expressive_high|>The author's blanket conversion broke the denoiser: values overflowed in an attention calculation before a later operation could normalize them. Even a more careful conversion looked acceptable for one window but diverged after several windows, because each generated chunk became history for the next.

Eris: <|sfx:sigh|>Uh, that first window would have reassured me. Which parts could the browser version actually keep at reduced precision?

Vestra: The chosen browser version keeps the text encoder and denoiser in full precision and uses mixed precision in parts of the decoder. <|prosody:expressive_high|>The lesson is to test a streaming model across continuations, not just one chunk.

Eris: The browser demo is a narrower interface than the Python demo: it supports changing text and playing generated motion, but doesn't expose the Python version's spatial constraints, waypoints or motion export. It uses ARDY's Core twenty-seven-joint skeleton. That skeleton is not a finished character, so the demo can map the movement onto a user-supplied VRM humanoid avatar. How does that mapping cope with a differently built body?

Vestra: <|prosody:expressive_high|>That mapping is called retargeting. Two characters may have different bone lengths, rest poses and even different numbers of spine joints. Copying raw rotations blindly can twist a limb or put a hand in the wrong place. Mini's code maps named source joints to avatar bones, accounts for the target hierarchy and orientation conventions, and scales the hips' movement using the relative heights of the rigs. It can fold a source spine segment into the corresponding target rotation. <|prosody:pause|>That makes a sensible mapping possible; it doesn't guarantee perfect contacts or prevent clothing and limbs intersecting on every avatar.

Eris: That contact caveat matters to me: I want the character to reach the chair we started with. And that returns us to evaluation. A score can compare generated movement with recorded movement, measure how far a hand misses its target, or count sliding feet. People can judge whether the result looks natural or fits the words. <|prosody:expressive_high|>None of those alone tells us everything: a smooth walk can miss the chair, and a hand can hit a target while the body moves awkwardly. The useful test depends on what the character needs to do and on the particular character receiving the motion.

Follow the research

Vestra: <|prosody:expressive_high|>Today’s separate news brief covers the day’s AI stories.