Teaching Video Models When to Speak
How TurnWise trains streaming video models to decide when to respond, how Memorilla builds question-specific memory, and how a separate predictor uses labeled history to make judgments. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio
Listen on Spotify · Watch on YouTube · Pocket Casts
Teaching a Video Assistant When to Speak
Eris: A training method called TurnWise helped video models decide when to answer and when to wait. In the researchers' tests, it beat competing training methods on the overall scores of three streaming video test suites, across two related Qwen models.
Vestra: The task is different from answering a question after watching a whole clip. A streaming assistant sees the video as it unfolds. At each decision point it can respond or stay silent, using only what it has seen so far and what it has already said. The tests check things like whether an answer is useful and timely, or whether the assistant responds when a particular event appears.
Eris: An early correct answer and a repetitive answer later can end up in the same conversation. If the whole conversation gets one good training score, <|prosody:pause|>the repetition can get rewarded along with the useful answer. A good follow-up can also get dragged down by an earlier mistake. <|prosody:expressive_high|>That bothers me: the useful follow-up deserves a chance.
Vestra: TurnWise changes the credit used during reinforcement learning, where the model tries multiple responses and updates its behavior using scores. For each decision, the researchers score the earlier part of the interaction as if the assistant stayed silent afterward. They subtract that reference from the score for the completed interaction, then compare the remaining returns across sampled conversations. That reduces the influence of earlier successes or failures on the later decision's training signal.
Eris: <|emotion:enthusiasm|>I like that the reference is something the assistant can actually do: <|prosody:pause|>wait. <|prosody:expressive_high|>But the later signal still includes what happens after that decision, doesn't it?
Vestra: It does. It's a score for the sampled continuation from that point, not a measurement of every possible alternative. <|emotion:enthusiasm|>I like the reuse here. The method reuses scores already calculated for the sampled replies, so the authors say it needs no extra model rollouts or new judge calls to calculate those references.
Eris: But how do they balance a whole answer against a short wait action?
Vestra: There's a second change. A text answer has many more generated pieces than the short wait action. <|prosody:expressive_high|>The researchers average the training loss within each decision first, then across decisions, so a long answer doesn't get more weight just because it's longer.
Eris: The comparison matters here. On one model, they tested the two changes separately. <|emotion:relief|>Combining them reduced the measured repetition compared with using turn-specific credit alone and improved the overall task scores. <|prosody:expressive_high|>That doesn't mean it eliminated repetition in every setting.
Vestra: And the score itself shapes what the model learns. The training reward combines answer quality and timing with penalties for repetition and answering outside annotated windows; an AI judge supplies part of the assessment. <|prosody:pause|>These were controlled video tests on two related model backbones, not a deployed assistant trial. The result is evidence for this training design under those conditions, not proof it will know when to speak in every live situation.
Memorilla's Question-Specific Memory
Eris: Memorilla is a memory system that gives a language model a compact readout tailored to the question being asked. Its researchers report the best average performance among their comparisons across eight personalization and question-answering tests, while giving the answering model just sixteen memory vectors instead of the source documents as text.
Vestra: The distinction is where the rest of the information lives. A separate encoder converts each document or conversation chunk into a numerical representation called an embedding. Those embeddings stay in a store. <|emotion:enthusiasm|>When a question arrives, a trained memory module uses the question to read across the stored embeddings and produce sixteen new vectors, which go into the answering model alongside the question. <|prosody:expressive_high|>The vectors are numerical inputs, not sixteen readable words.
Eris: So a question about someone's food preferences could draw on several past conversations, while a question about their travel plans produces a different readout from the same store. <|emotion:enthusiasm|>I like that the question gets a say in what comes back. <|prosody:expressive_high|>But the store itself grows as documents are added; the entire history hasn't been squeezed permanently into sixteen tokens.
Vestra: Right. The researchers train the memory module to help a frozen language model produce target answers, while leaving the encoder and answering model's weights unchanged. They compare it with full document context, text retrieval and other memory and compression systems. <|prosody:pause|>It leads on average, but not on every task: on their biomedical question-answering test, several baselines do better.
Eris: And fewer input tokens can matter for speed. <|emotion:surprise|>In one test with hundreds of documents, the authors report that a query using Memorilla was <|prosody:pause|>about twenty-seven times faster than feeding the full document collection to the answering model. <|prosody:expressive_high|>That was one specific setup on a single high-end GPU, with short generated answers. Document encoding was a separate upfront cost, and the full-context timing depended on the serving setup. Did the memory module and the baselines get the same task-specific training?
Vestra: <|sfx:humming|>Hmm, there's another comparison wrinkle: the memory module was trained on the training splits of the evaluation tasks, while some baselines used their standard configurations. The result doesn't isolate the architecture from the benefit of that supervision. <|prosody:pause|>And the authors warn that the compressed readout can lose exact details, such as specific names.
Eris: <|prosody:expressive_high|>Losing a specific name would worry me in a system meant to remember someone's preferences. The team also tried memory in a text adventure, where an agent explores rooms and uses objects to complete a goal. The trained memory improved average completion compared with the same frozen agent given its history without the memory module. Did the vectors replace the history in that experiment?
Vestra: <|emotion:contemplation|>That experiment tests a different use of memory. Both agents kept the textual history in the prompt; the trained module added eight vectors on top. <|prosody:expressive_high|>So the result suggests that a learned readout helped the agent use its history, not that eight vectors replaced the history. The held-out game set was small, and the longer games remained difficult.
Learning Decisions From Labeled History
Eris: A system called Decoupled Predictive Decision Making improved benchmark judgments by combining a language model with a separate predictor trained on structured data. When researchers gave it and ordinary prompting the same labeled examples and questions, the combined system achieved a higher average accuracy across fifty benchmarks with the larger Qwen model they tested.
Vestra: Think of a decision about whether a proposed tool call is appropriate. The system has examples of earlier cases and their correct judgments. A language model turns each case and its candidate answers into numerical features. A separate predictor receives those features, the historical labels, and the current case; it estimates which answer is most likely. A decision rule chooses the highest-probability answer.
Eris: The division of labor is the point: the language model represents what the cases mean, while the predictor uses the labeled examples to infer a relationship. <|emotion:enthusiasm|>If new labeled cases arrive, the system can use them as history <|prosody:pause|>without changing either model's weights. That's the part that excites me: being able to use new experience that way.
Vestra: <|emotion:enthusiasm|>I'm interested in how far that same predictor goes. The same predictor was used for several kinds of judgment, including tool use, answer preferences and checking reasoning steps. <|prosody:expressive_high|>In the main comparison, the combined system often had more historical examples than the language model could fit or finish processing in its prompt.
Eris: Then I'd be wary of attributing the whole gain to that division of labor. What happened when they got the same history?
Vestra: <|prosody:expressive_high|>In the equal-history comparison, the improvement remained, though it was smaller for the larger Qwen model than in the main comparison.
Eris: The history has to contain useful labels, though. If the examples are misleading or don't cover a new situation, the predictor can't simply invent the missing evidence. Did they test how much those labels matter?
Vestra: The authors tested that dependency by changing historical labels and comparing distinct examples with repeated ones. They also looked at reasoning-step judgments where most examples were correct. <|prosody:expressive_high|>Their balanced measure gave equal weight to recognizing correct and incorrect steps; the combined approach improved the average, though individual task and model results varied.
Eris: These were benchmark judgments against reference answers, not a live trial showing that a complete agent finishes more real-world jobs. <|prosody:pause|>The result is a way to use labeled experience for particular decisions; collecting reliable feedback and testing what happens when those decisions are chained together remain separate challenges.
Follow the Research
Eris: <|prosody:expressive_high|>Find the sources and follow every story from the show, every day, at Ground Truth, groundtruth dot day.
Vestra: <|emotion:enthusiasm|>Follow AI in Detail for more news and research. Which of these approaches would you want to see tested in a real application? Tell us in the comments.