AI papers — 2026-06-10

Problem: Processing hour-level videos with dense attention incurs prohibitive compute costs, and integrating heterogeneous agent tasks with video understanding causes catastrophic forgetting of foundational reasoning.

Method: A 30B Mixture-of-Experts model with only 3B active parameters uses a SigLIP-400M-based native-resolution ViT encoder and DeepSeek Sparse Attention (DSA) adapted for GQA-based models, reducing complexity from O(L²) to O(L·k) where k=2048. A four-stage training pipeline progresses from projector initialization through general pretraining at 32K context (~1T tokens), multi-task injection at 64K context (~2T tokens, covering OCR, math, GUI, grounding, coding, tool-use), and long-context extension to 256K tokens. Post-training employs supervised fine-tuning with synthetic chain-of-thought, specialized RL modules per domain (grounding, spatial, math, counting, OCR), and a "Cross-Modal Multi-Teacher On-Policy Distillation" scheme consolidating 13 domain-specific teachers via token-level overlap estimation to prevent forgetting.

Key Results:

• LongVideoBench: 74.1 vs. 70.5 for Qwen3-VL-235B
• LiveCodeBench v6: 64.2; τ²-Bench (tool-use): 82.6
• DSA reduces prefill cost by 3× and decode cost by 5× vs. dense attention at 128K context
• Video-MME-v2 accuracy (512 frames): 42.4; ActivityNet-TimeLens mIoU: 58.5

Why it matters / Caveats: This is the first successful integration of DSA with GQA-based multimodal models, enabling practical 256K-context hour-level video processing; the multi-teacher distillation scheme addresses a real deployment bottleneck. Evaluation is benchmark-centric and the multi-stage training complexity (13 specialized teachers) may limit reproducibility.

Problem: Traditional photogrammetry and LiDAR reconstruction of planetary-scale 3D environments incur extreme acquisition costs and processing latencies that make global coverage infeasible.

Method: A native 3D Gaussian Splatting (3DGS) generative model is trained directly on real-world urban reconstructions rather than synthetic assets, operating in compressed 3DGS space to preserve non-manifold topologies. The ABot-3DGS Reconstruction Engine converts multi-source imagery (satellite, aerial, urban) into 3DGS scenes via a hierarchical block-based architecture with semantic-aware optimization. A multi-LOD decoder produces hierarchical outputs across 6 LOD levels (zoom 14–19), and a sliding-window inference strategy enforces spatial coherence across 1.6 km × 1.6 km tiles. VLM-based conditioning provides cross-domain adaptation for satellite imagery robustness, and a global production pipeline processes 312,500 tiles over a 1000-GPU cluster.

Key Results:

• FID score: 16.1 vs. 69.5 (EarthCrafter), KID: 0.006 vs. 0.061
• Generation rate under 10 minutes per km² vs. months-to-years for photogrammetry
• Coverage: 300+ cities, 190+ countries; 3.2 trillion Gaussian primitives
• Human evaluators rated ABot-Earth higher than Google Earth on "Overall Aesthetics"

Why it matters / Caveats: This is the first end-to-end generative pipeline for interactive, streamable planetary-scale 3D environments from satellite imagery, enabling applications in UAV navigation and embodied AI. It currently generates only aerial-level (~200 m) detail; street-level generation and closing the quality gap to hand-optimized reconstructions are stated future work.

Problem: LLM agents are limited by static training environments and inefficient interaction feedback, yet existing co-evolution approaches either keep the environment fixed or require separate auxiliary models, increasing deployment complexity.

Method: A single LLM simultaneously acts as agent and environment through two coupled modules. World-In-Agent (WIA) generates future state predictions after each action over horizon H; alignment between predicted and actual states generates predictive rewards via Longest Matching Subsequence (LMS) that modulate task rewards multiplicatively as ℛₜ = ℛₜₐₛₖ(1 + ℛₚᵣₑ). Agent-In-World (AIW) analyzes failed trajectories to extract labeled failure modes (e.g., ENTITY_CONFUSION, WRONG_TARGET_LOCATION) and retrieves historically similar failures to reshape the training distribution toward hard cases. Training combines state-level advantages Aˢ and trajectory-level advantages Aᴱ via a mixing coefficient α, optimized with GRPO-style policy gradients and KL divergence regularization.

Key Results:

• ALFWorld (Qwen 2.5-1.5B): 90.9% success vs. 86.7% GiGPO baseline (+4.2%)
• WebShop: 71.9% vs. 65.0% GiGPO (+6.9%)
• Search-augmented QA: 45.8% average, +8.2% on 2WikiMultiHop vs. GiGPO

Why it matters / Caveats: Demonstrates that a single LLM can serve as both agent and environment to achieve practical co-evolution without auxiliary models, yielding consistent gains across diverse task types. Evaluation is restricted to text-based environments, and state grouping relies on fixed similarity thresholds that may limit cross-task generalization.

Problem: Existing harness optimization methods for LLM agents require ground-truth validation sets, which are difficult to acquire in practical deployment settings.

Method: Retrospective Harness Optimization (RHO) improves agent harnesses using only past trajectories through a three-stage self-supervised loop. A Determinantal Point Process (DPP) selects k=10 difficult yet diverse coreset tasks from historical trajectories (balancing difficulty scores and embedding similarity with weight θ=0.7). G=3 parallel rollouts per coreset task extract two diagnostic signals: self-validation (incorrect tool invocations, false assumptions) and self-consistency (contradictions across parallel attempts). N=3 candidate harnesses are then proposed conditioned on these diagnostics and evaluated via pairwise agent self-preference ranking, accepting only those with mean score above zero. The backbone agent is Codex (GPT-5.5 with high reasoning effort) and the harness is represented as a filesystem directory of scripts, markdown instructions, and skills.

Key Results:

• SWE-Bench Pro: 59% → 78% (+19 percentage points absolute)
• Terminal-Bench 2: 71% → 76% (+5 points)
• GAIA-2: 29% → 37% (+8 points)
• RHO achieves 0.78 vs. 0.62 for Meta-Harness (a validation-feedback baseline) on SWE-Bench Pro

Why it matters / Caveats: Shows that agents can improve their own operational scaffolding without external grading, outperforming even baselines that use held-out labels. Requires environments tolerating repeated attempts and pairwise self-preference is a noisy estimator, occasionally selecting suboptimal harness candidates.

Problem: Finite model context windows create a hard ceiling for long-horizon research tasks, and existing passive strategies like summarization or truncation lack the planning needed to effectively decompose and delegate complex multi-step information gathering.

Method: A main-distributes, sub-executes architecture where the main agent coordinates via a call_sub_agent tool and subagents operate in independent 64K-token contexts equipped with search, visit, google_scholar, and python tools. Harness design follows four principles: encourage delegation, provide comprehensive briefings (including rationale, established facts, open gaps, tried directions), retain core judgment at the main agent, and require citation-grounded subagent reports. Training data is collected from RedSearcher/OpenSeeker datasets using two configurations (same model for both roles; stronger main + weaker sub), filtered to retain only trajectories with correct answers and remove hallucinated citations or tool misuse. The base model is Tongyi DeepResearch-30B-A3B, fine-tuned with next-token prediction loss masking all environment returns, at batch size 128 and cosine-decaying learning rate from 5e-5 to 1e-6.

Key Results:

• BrowseComp: 68.1 vs. 67.9 best prior 30B-scale model; GAIA: 82.5 (+2.2)
• xbench-DeepSearch: 80.8 vs. 78.0 LongSeeker (+2.8)
• +24.7-point absolute gain over untrained base model on BrowseComp (43.4 → 68.1)
• Harness alone (without training) adds +10.0 points over the base framework on BrowseComp

Why it matters / Caveats: Demonstrates that delegation intelligence—decomposing and offloading multi-step retrieval—can be internalized into model weights and generalizes to single-agent settings. Only single-level delegation is supported (subagents cannot further delegate), and transfer to non-research long-horizon tasks remains unstudied.

Problem: Vision-Language Models cannot effectively process hours-long videos due to token explosion and attention dilution when handling full-length visual sequences.

Method: MemDreamer separates perception from reasoning via two components. A Hierarchical Graph Memory uses a three-tier top-down architecture for semantic abstraction, with a foundational graph capturing spatiotemporal and causal relations built incrementally from streamed video input. An Agentic Retrieval Mechanism operates at inference time using an Observation-Reason-Action loop that navigates the memory hierarchy, searches nodes, and traverses logical edges to retrieve only the most relevant evidence. The system is designed as plug-and-play, requiring no retraining of the underlying VLM.

Key Results:

• State-of-the-art across four long-video benchmarks with only a 3.7-point gap to human expert performance
• 12.5-point absolute accuracy gain over prior methods
• Reasoning context window constrained to approximately 2% of full-context ingestion
• Establishes a positive linear correlation between VLM logic-reasoning performance and long-video understanding

Why it matters / Caveats: Introduces agentic capability scaling as a new paradigm for multimodal comprehension, dramatically reducing computational overhead while approaching human-level performance. The abstract does not explicitly discuss failure cases, scalability limits beyond the tested benchmarks, or generalization to video domains outside the evaluation set.

Problem: Prior character animation methods rely on intermediate representations such as pose skeletons or masked backgrounds that introduce information loss and restrict task flexibility.

Method: Built on a 14B Wan2.1 I2V backbone with LoRA adapters, SCAIL-2 directly concatenates driving video latents instead of extracting pose estimates, and adds 4(K+1) channels for In-Context Mask Conditioning (K=6 binding slots plus an environment switch) to distinguish animation tasks from character replacement. Mode-Specific Shifted RoPE applies different temporal and spatial position encodings for each mode. The MotionPair-60K dataset (59,376 synthetic motion pairs generated via SCAIL, Wan-Animate, and MoCha using agentic editing loops) is constructed using a Reverse Driving strategy where synthetic video becomes the driving input and the original becomes the denoising target. Post-training uses Bias-Aware DPO with hand-region masking to correct pose estimation errors. Training runs 3,500 pretraining steps plus 400 DPO steps on 64 H100 GPUs.

Key Results:

• Human evaluation outperforms open-source baselines (SCAIL, Wan-Animate) on motion accuracy and identity consistency, approaching proprietary Kling 3.0
• Pose-driven metrics (Table 2): SSIM 0.6453, PSNR 19.09, LPIPS 0.2231, FVD 287.11
• Video-Bench Imaging Quality: 4.43 vs. SCAIL's 4.25; Appearance Consistency: 4.38
• Demonstrates zero-shot multi-character identity isolation without dedicated multi-character training data

Why it matters / Caveats: Unifies single/multi-character animation and character replacement under one end-to-end framework with no intermediate representation bottleneck. The approach has strict dependence on large-scale, high-quality paired training data, and fine-grained regions (fingers, facial expressions) require explicit preference optimization.

Problem: Applying PPO-style ratio clipping to flow matching models for image/video generation produces noisy, single-sample trust region estimates that over-constrain some trajectory regions while under-constraining others, causing reward hacking and instability.

Method: Flow-DPPO replaces ratio clipping with a divergence-based asymmetric mask exploiting the closed-form KL divergence available for Gaussian per-step policies in flow models: DKL(πθold‖πθ) = ‖μθold − μθ‖²/(2σ²), computed at zero additional cost. Updates are blocked only when gradients push the policy away from the old policy AND KL divergence exceeds threshold δ; corrective updates moving back toward the old policy are never blocked. The training objective is ℒFlow-DPPO(θ) = E[∑(Mᵢₜ · rᵢₜ(θ) · Âⁱ − β·DKL)] where the mask Mᵢₜ selectively gates gradients. Evaluated on Stable Diffusion 3.5 Medium, FLUX2-klein-base-9B, and FLUX.1-dev using GenEval2 (20k synthetic prompts) and PickScore.

Key Results:

• GenEval2 score on FLUX2-9B: 57.7 (Flow-DPPO) vs. 46.8 (Flow-GRPO) and 49.0 (GRPO-Guard)
• PickScore: 25.76 vs. 25.61 (Flow-GRPO)
• KL retention: 0.49×10⁻³ vs. 0.79×10⁻³ (Flow-GRPO)
• Maintains consistent gains under multi-epoch rollout reuse (G32-I2, G64-I2) where baselines plateau or degrade

Why it matters / Caveats: Provides theoretical justification (Theorems 1–2) for trust-region methods in flow models and demonstrates that exact divergence computation substantially outperforms noisy ratio-based constraints. Evaluation is limited to image generation; applicability to video or other modalities, and guidance on divergence threshold δ tuning, remain open.

Problem: AI systems that generate data journalism lack mechanisms to ensure published claims are traceable back to evidence sources, and most produce text-heavy articles rather than engaging multimodal formats.

Method: The Data2Story system implements a "virtual newsroom" of seven specialized agents built on Claude Opus 4.7: Detective (web search augmentation), Analyst (Python code execution with line-level provenance), Editor (ranking findings and editorial planning), Designer (multimedia modality selection), Programmer (HTML rendering), Auditor (visual/structural defect identification), and Inspector (binding every visible claim back to supporting evidence—code lines, datasets, or URLs—via claim decomposition into partial findings with explicit source links).

Key Results:

• Human evaluation (n=53): Data2Story overall mean 4.21/7 vs. human articles 3.38/7; 39 of 53 reviewers preferred Data2Story
• Largest advantage on "Transparency" dimension: +1.49 points over human articles
• 93% of Data2Story claims have traceable evidence bindings vs. ~25% for human articles
• Human-in-Agent coverage 50.4%; Agent-in-Human coverage 35.1%

Why it matters / Caveats: The Inspector mechanism directly addresses AI journalism's reproducibility gap, making claims auditable in a way that even carefully crafted human articles rarely achieve. Human journalists retain advantages in editorial framing and creative design, and performance degrades on long-form design-intensive pieces; potential training-data contamination on well-known articles is acknowledged.

Problem: Existing video generation benchmarks focus on visual quality and semantic alignment while leaving physical plausibility, 3D consistency, and long-horizon interaction control largely unassessed.

Method: WorldOlympiad evaluates eight video generation pipelines on 1,000 videos (400 robotics from RoboCOIN, 400 gaming from GameGen-X, 200 real-world from LVD-2M) across three tracks: a Physical Track using MLLM-as-judge with SAM3-based object segmentation to assess mechanics, thermodynamics, and material properties; a Geometry Track reconstructing videos via Gaussian splatting with Depth Anything 3 and measuring reconstruction quality (Srecon), meta-view quality (Smeta), and camera-trajectory alignment (Straj); and an Interaction Track using hierarchical MLLM judging at chunk, transition, and global levels with CLIP-based semantic grounding (Sinteract = (1−λ)Smllm + λS̃clip, λ=0.1). A three-stage annotation pipeline using Gemini-3-Pro-Preview generates task captions from chunked video.

Key Results:

• Top model (LingBot-World): overall 0.683; Physical 0.942, 3D Consistency 0.373, Interaction 0.734
• Even top models achieve only 0.424 on 3D consistency, revealing an unresolved geometry-simulation gap
• Human preference alignment shows strong Spearman correlation ρ=0.95 with automatic scoring

Why it matters / Caveats: Provides a unified multi-domain evaluation protocol revealing systematic gaps between perceptually plausible and physically/geometrically sound generation. Equal-weight metric aggregation may not reflect application priorities, and MLLM judge reliability depends on model calibration.

Problem: PPO-style importance-ratio clipping is poorly calibrated for LLM training because small probability increases on low-probability tokens produce very large ratios while changing little probability mass, destabilizing trust regions particularly under low-precision conditions.

Method: Divergence Regularized Policy Optimization (DRPO) replaces PPO's hard clip with smooth advantage-weighted regularization using a binary total variation (TV) constraint on absolute probability shift rather than importance ratios. The gradient weight is defined as wₜ = 1 − sign(Aₜ(rₜ−1)) × |π(yₜ|sₜ) − μ(yₜ|sₜ)|/δ, where advantage weighting |Âₜ| keeps trust-region boundaries stable and independent of reward scale. The regularizer is an advantage-weighted ℓ₂² penalty on probability differences. Experiments use three Qwen models (4B, 30B, 35B) and DeepSeek-R1-Distill on AIME 2024/2025 math benchmarks.

Key Results:

• DRPO achieves competitive or superior accuracy across all six model/benchmark settings vs. PPO and DPPO baselines
• Particularly stable under FP8 low-precision training where ratio-based methods collapse
• Faster convergence than DPPO with hard masking; δ=12.5 performs well universally across settings

Why it matters / Caveats: Identifies gradient-induced geometry as more important than the nominal form of the divergence penalty, and provides bounded gradient weights [1−1/δ, 1+1/δ] that prevent unbounded scaling on rare tokens. Evaluation is limited to math reasoning with rule-based verification, and the interaction between advantage weighting and other RL components is not fully characterized.

Problem: Diffusion-based lip synchronization models achieve high visual quality but require full-sequence bidirectional attention and many denoising steps, making real-time deployment at playback frame rates impractical.

Method: A 14B bidirectional teacher model is distilled into causal autoregressive students (1.3B and 14B) that operate with only two denoising steps using ODE indices (0, 30). Trajectory analysis of the teacher reveals a CFG fidelity–sync tradeoff: Sync-Window DMD applies classifier-free guidance (scale 4.5) only within denoising steps 20–40 (the sync-favoring band) and no-CFG (scale 1.0) elsewhere. A SyncNet-Based Reward adds explicit lip-sync supervision via multiplicative gradient weighting w(x̂₀) = exp(β·R(D(x̂₀), a)) with β=2. Training proceeds in two stages: pretraining via Diffusion Forcing on real data (VoxCeleb2, Hallo3, HDTF), then distillation using Self Forcing DMD with the trajectory-derived modifications.

Key Results:

• 1.3B student: 31.58 FPS (17.6× faster than bidirectional 1.3B; crosses 25 FPS real-time threshold); TTFF 0.32 ms
• HDTF FVD: 118.86 (1.3B), 107.88 (14B), best among diffusion baselines
• HDTF CSIM: 0.943 (1.3B), strong identity preservation
• User study MOS: 4.32 naturalness for 1.3B, exceeding LatentSync (3.53) and X-Dub (3.97)

Why it matters / Caveats: First autoregressive diffusion method for video-to-video lip sync enabling true streaming deployment; the 1.3B model crosses the real-time threshold at 31.58 FPS. Sync-C metrics trail specialized CFG-optimized baselines, analysis was derived from only 10 held-out clips, and the system is fixed to 512×512 resolution and 81-frame sequences.

Problem: Single-dataset test-time prompt learning methods suffer from cross-dataset interference in real-world deployments with heterogeneous task streams, where optimizing prompts for one domain degrades performance on others.

Method: EEVEE jointly optimizes a Router R and a set of K specialized prompts 𝒫={p₁,...,pₖ} through a three-stage co-evolution procedure. Router Evolution samples and mutates routing candidates, reflects on failure cases, and selects improvements via downstream accuracy, routing consistency, and balance metrics. Prompt Evolution applies mutation and reflection on routed data, maintaining Pareto-front pools to preserve complementary prompts and accepting candidates only if they improve over the empty-prompt baseline and remain on the Pareto frontier. The three stages are Initialization (diverse coverage via greedy selection), Exploration (interleaved router/prompt evolution under annealing weights), and Convergence (fixed router, concentrated optimization). Evaluated on GPQA Diamond, Formula, TheoremQA, and HumanEval using Qwen3-4B-Instruct and DeepSeek-V3.2.

Key Results:

• Qwen3-4B: average 51.75 (+10.38 over baseline), +14.02 over GEPA, +16.83 over ACE
• DeepSeek-V3.2: average 64.07 (+24.32 over baseline), +8.24 over GEPA
• Multi-benchmark retention: +41.53 cumulative vs. baselines that accumulate negative retention losses
• Single-benchmark HumanEval: 73.17 (Qwen3-4B), 92.82 (DeepSeek-V3.2)

Why it matters / Caveats: First framework addressing multi-dataset test-time prompt learning, showing that router-prompt co-evolution can prevent cross-domain interference while preserving single-benchmark competitiveness. Requires ground-truth labels for feedback, stochastic search prevents exact reproduction, and GPQA Diamond regressions suggest prompt learning underweights domain knowledge.

Problem: Memory-augmented QA systems consume excessive tokens when passing raw text and image evidence to language models, with images alone expanding to hundreds of visual tokens per item.

Method: A small LLM/VLM Compressor encodes each evidence item into a single latent token via a learnable [MEM] embedding; an MLP Retrieval Module projects latent tokens and queries into a shared retrieval space for similarity-based selection; and a frozen LLM/VLM Generator receives the projected latent tokens as continuous evidence. An optional Reconstruction Decoder recovers text autoregressively and predicts CLIP embeddings for images. Training jointly optimizes three losses: Reconstruction Loss (autoregressive text decoding + CLIP embedding prediction), Contrastive Loss (multi-positive InfoNCE aligning queries with supporting latent tokens), and Distillation Loss (KL divergence between generator outputs on raw evidence (teacher) vs. latent tokens (student)). Only compressor LoRA, decoder LoRA, and two projection matrices are trainable; the generator is frozen throughout.

Key Results:

• Text QA at k=5: ~3× token efficiency over RAG baselines (71 generator tokens vs. 208–209); out-of-domain Recall@5: 52.2
• Multimodal QA (WebQA, LLaVA-13B): 69.4 F1 at k=5 with 82 tokens vs. 53.0 Nemo-Emb using 1885 tokens (~10× token efficiency)
• HotpotQA-trained model shows strong out-of-domain performance on MuSiQue and 2WikiMultihopQA without additional tuning

Why it matters / Caveats: Demonstrates a practical unified latent representation for both retrieval and generation across text and image modalities, enabling deployment on edge devices. The design assumes atomic, independent evidence units and struggles with complex structures requiring spatial, temporal, or layout relations such as tables or videos.

Problem: Existing GUI agent benchmarks focus on short-horizon, general-purpose tasks and do not systematically evaluate whether agents can execute long-horizon, domain-specialized, economically meaningful professional workflows.

Method: Workflow-GYM constructs 338 tasks across 6 professional domains and 23 subdomains (Easy: 30–44 steps, 38.2%; Medium: 45–60 steps, 47.0%; Hard: 61–110 steps, 14.8%) sourced from 71 domain experts and filtered for realism, specificity, complexity, and verifiability. Each task runs in one of 58 virtual machine base environments with professional software pre-installed; evaluation uses automated rule-based or LLM-based assessment on final artifacts. A three-stage validation pipeline performs environment verification, instruction-level LLM-as-judge evaluation, and end-to-end agent testing. Agent performance is measured with Pass@1 and Pass@3 across difficulty tiers.

Key Results:

• Best model Gemini-3.1-Pro: 30.67% Pass@1, 41.12% Pass@3
• Kimi-k2.6: 29.68% Pass@1, 41.42% Pass@3; GPT-5.4: 17.85% Pass@1
• Textual procedure hints: +2.76% to +12.23% improvement; video tutorial hints: additional +3 to +7 tasks on 100-case subset
• Strong negative correlation (r=−0.97) between workflow incompletion and model performance; over 100 repetitive action looping cases identified

Why it matters / Caveats: Reveals a fundamental mismatch between continuous human interaction and the discrete observation-action paradigm in current agents, and identifies key failure modes (error propagation, objective drift, software knowledge deficiency). Evaluation relies on snapshot-based observation, and models struggle with fine-grained spatial manipulation in complex creative software.

Problem: Chain-of-thought supervised fine-tuning (CoT-SFT) catastrophically degrades long-context retrieval in hybrid linear-attention models while improving reasoning, e.g., HypeNet-9B on NIAH-S2@256K drops from 67.2% to 9.4% after CoT-SFT.

Method: The authors model CoT reasoning as a latent Markov process and prove (Theorem 4.6) that gradient magnitudes on attention logits decay exponentially with token distance, causing query-key projections (WQ, WK) to drift toward short-range patterns while value projections (WV, WO) receive uniform gradients (Theorem 5.1, routing-extraction gradient decoupling). The fix, QK-Restore, is a training-free method that restores only WQ and WK from pre-SFT checkpoints while preserving post-SFT WV and WO, thus recovering long-range routing without sacrificing reasoning gains. Base model is HypeNet trained on FineWeb-Edu (80%) + UltraData-Math (20%); CoT-SFT uses MiroMind-M1.

Key Results:

• HypeNet-5B: NIAH-S3@256K improves from 65.4% to 76.4% using QK-Restore
• HypeNet-2B: NIAH-S3@128K recovers from 7.6% to 30.2% (+19.6 points)
• Math reasoning largely preserved: MATH500 drops only −0.6 points on HypeNet-5B
• Non-CoT instruction tuning preserves retrieval, confirming the failure mode is CoT-specific

Why it matters / Caveats: Identifies a critical and previously unknown failure mode in post-trained hybrid models and provides a zero-cost training-free fix applicable across architectures. Theory assumes CoT-Markov structure and reversibility; applicability to pure softmax transformer models is limited, and analysis focuses primarily on mathematical reasoning.

Problem: Little is understood about the internal representations of text-to-speech language models where text prefix tokens and generated speech tokens share a single residual stream, and existing interpretability methods designed for text-only LMs do not account for this mixed-modality scenario.

Method: BatchTopK sparse autoencoders (dictionary size d=16,384, k=50 active features per token) are trained on ~250M tokens from the Emilia dataset over the residual stream of CosyVoice3 (Qwen2.5-0.5B backbone, 896 hidden dimensions, 28 layers). A Modality-Aware Auto-Interp pipeline categorizes features by activation location using thresholds—audio-modal (≥0.8 speech fraction), text-modal (≤0.2), mixed (otherwise)—then uses Gemini 3.0 Pro to generate descriptions from modality-specific evidence (marked token windows for text; 1-second audio clips for speech). Steering interventions are applied through the SAE latent space with frozen encoder/decoder and controlled feature activation perturbations.

Key Results:

• Auto-Interp AUROC (Layer 20): text-modal 0.921, audio-modal 0.653, mixed 0.558
• Laughter probability steered from 0.02 → 0.791 at α=+60
• Speaker gender (wav2vec2 P(male)) shifted from 0.629 to 0.944 (α=−50) or 0.063 (α=+50)
• Speech rate voiced duration: 3.96 s → 10.57 s (slower, α=−50) or 2.75 s (faster, α=+50)
• Layers 16–20 dominated by audio-modal features (76.1% at layer 16); layer 23 reverts to 83.1% text-modal

Why it matters / Caveats: Demonstrates that sparse features in TTS backbones are both interpretable and causally steerable, enabling fine-grained synthesis control (prosody, gender, rate, laughter) and providing mechanistic insight into cross-modal processing across layers. Analysis is limited to a single small model (CosyVoice3-0.5B), there is a circular evaluation risk from using the same Gemini model for labeling and scoring, and sub-token temporal resolution is limited to the 25 Hz speech-token rate.