AI papers — 2026-08-04
Jump to one of 30 papers
- SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks
- LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
- VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
- Progressive Agent Skill Generation via Reinforcement Learning
- DAPD: Dual-Anchored Policy Distillation
- UEmbed: Unified Sparse and Dense Multimodal Embeddings
- WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity
- CADENA: Stepwise CAD Reverse Engineering
- SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
- SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
- Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs
- WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
- Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representations
- GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning
- Roomer: Reflective Object-Grounded Model Editing and Repair for 3D Indoor Layout Synthesis
- GPTQ-2D: Cubic-Time Two-Sided Adaptive Rounding
- DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents
- 3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering
- DiffusionGemma Technical Report
- LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation
- StyleForge: Indoor Furniture Styling by Counterfactual Reasoning in a Hypergraph Field
- DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents
- Poplar: A Scalable Pipeline for Human-Centric Image Dataset Synthesis
- ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step
- Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts
- RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems
- Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
- ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures
- A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples
- GEOID-Flood: A Large-Scale Multi-Modal Benchmark Dataset for Flood Segmentation
SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks →
Technical breakdown
Problem: Media production workflows need a single TTS/audio system that supports both zero-shot voice cloning from reference audio and fully instruct-driven (caption-only) multi-speaker speech and audio generation, including environments, personas, and local audio effects, but existing instruct TTS systems generate only speech, decompose speaker control into rigid attribute labels, and typically don't preserve zero-shot capability in the same model.
Method: The pipeline pairs a data system, SwanData-Caption (four stages: coverage design, SwanData-Speech preprocessing, caption annotation with a style-persona library, and data refinement via waveform filtering/human auditing), with SwanTale, a model built on SwanVAE (a 48kHz-to-25Hz, 96-dim continuous-latent autoencoder with a decoder-side Transformer Resampling Block and auxiliary flow-matching/causal-prediction/chroma-energy alignment losses) feeding a non-causal flow-matching DiT. The DiT adds Engram conditioning (an n-gram hashed memory layer for recurring caption patterns), reward-conditioned quality control (STOI/PESQ/SI-SDR/MOS scores injected as a controllable quality caption/flag), and Unified MoE (task router for sample-level shared experts plus a diffusion-time-aware dynamic Top-P audio router with null/skip experts). Training uses a four-stage curriculum (zero-shot base → dense caption adaptation → full MoE caption-mixture → high-quality SFT) followed by GRPO-based reward post-training (via a Flow-GRPO SDE conversion) targeting pronunciation, stability, and speaker-attribute control.
Key results:
- SwanVAE: best PESQ (4.1683) and MCD (0.9638) on speech reconstruction among compared codecs/autoencoders; leads on singing voice PESQ/STOI/MCD; achieves 407.0M parameters (51.7M encoder, 355.0M decoder).
- Zero-shot monologue TTS (SwanBench-Speech): SwanTale reaches Timbre Consistency 0.95, Content Error 0.086, SpeechJudge 3.75, Expressive Richness 3.90, Expressive Hierarchy 3.70 — improving over its SwanVoice base on all these metrics.
- Instruct TTS (InstructTTSEval): ranks first on Chinese APS (86.1), ties first on English APS (84.2).
- SwanBench-Scene: highest overall Mean MOS of 4.22 across advertising, comic drama, and general-scene categories.
- Ablation on SwanBench-Caption: removing Unified MoE drops Instruction Accuracy from 3.39 to 3.02 and Overall Expressiveness from 3.82 to 3.56; scaling the caption encoder from 8B to 32B raises these to 3.70 and 3.98.
Why it matters / caveats: Demonstrates that one model can jointly serve zero-shot and instruct multi-speaker/multi-modality audio generation at production quality, but the paper notes remaining weaknesses in Role-Play instruction following and English descriptive-style directives, and some baselines still beat it on raw content accuracy/sound fidelity in specific settings.
LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks →
Technical breakdown
Problem: Existing LLM agent harnesses maintain task execution, task state, and completion assessment all within one continuously growing context, which makes state hard to track and lets an agent's incorrect self-assessment of progress silently corrupt later decisions on long-horizon tasks.
Method: LongHorizon-Harness reformulates long-horizon execution as explicit task-state management via a Manage-Execute-Audit (MEA) loop: a manager maintains a structured task state (requirements/artifacts/facts) and defines one bounded subtask contract per round; a fresh-context executor (GUI or CLI, e.g., Claude Code, Codex CLI, OpenClaw) performs only that subtask with a budget-bounded episode, after which its raw trajectory is discarded; a read-only auditor independently inspects the resulting environment (never trusting the executor's own completion claim) and returns an audit report that is the only information carried across rounds. A lightweight AgentAdapter interface lets model and harness backends be swapped in for each of the three roles without modifying their native agent loops; execution is capped at 1800s per executor round, 300s for manager/auditor, and a max of 25 MEA rounds.
Key results:
- WeaveBench (114 tasks): with Qwen 3.7-Plus + Claude Code, PassRate improves from 51.8% to 80.7% (mean task score 0.702 to 0.835), exceeding the officially reported best (Claude Opus 4.7 + Claude Code) of 41.2%.
- Terminal-Bench 2.1: improves from 69.7% to 77.2% with Qwen 3.7-Plus; reaches 83.1% with Codex + GPT-5.6 Luna.
- OSWorld 2.0 (108 tasks): binary completion rises from 2.8% to 8.3% and partial score from 21.5% to 35.2% with Qwen 3.7-Plus; on a 34-task Opus 4.7 subset, binary completion rises from 20.6% to 35.3% (partial 55.8% to 66.9%).
- Token overhead varies by task: 2.3x baseline tokens on WeaveBench, 3.6x on OSWorld 2.0, but 24% fewer tokens on Terminal-Bench 2.1 while still achieving higher success; auditor accounts for 19.4–38.1% of total tokens across benchmarks, manager only 2.0–8.1%.
- WeaveBench Games subset (17 tasks): mean score for Qwen improves from 0.524 to 0.733 (exceeding Opus 4.7's baseline 0.680), and for Opus 4.7 from 0.680 to 0.809.
Why it matters / caveats: Shows agent capability is a joint model-harness property — a weaker model with this harness can outperform a stronger model without it — but gains are smaller on tasks whose bottleneck is an individual model capability (e.g., visual perception, coding) rather than long-horizon state tracking, and token cost can rise substantially (up to 3.6x) depending on task/model.
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation →
Technical breakdown
Problem: In multimodal on-policy distillation (OPD), privileged-view teacher corrections used to supervise a student are "source-mixed" — combining true visual-evidence signal with linguistic priors and teacher-specific artifacts — so distilling the full correction (or merely reweighting it by visual advantage) transfers non-visual and even misleading signal to the student.
Method: Visual Attribution Distillation (VAD) queries the same fixed (frozen) teacher on the same student-generated prefix under an evidence-present crop and an evidence-removed/degraded crop, and computes the shift in centered log-probabilities ut as a signed intervention-derived proxy for the visual evidence direction. It then does a one-sided regularized projection of the full teacher correction rt onto ut to obtain a visually-attributed component rt_vis and residual rt_res, refines this into separately budgeted support/refutation branches (rt_VAD), and reconstructs a student-anchored target qT,t_VAD by adding rt_VAD to the student's own current distribution (rather than matching the teacher directly). Training minimizes a Jensen-Shannon divergence to this reconstructed target plus a weak privileged-teacher regularizer (weighted by how much of the correction is visually unexplained); no extra teacher calls or views are needed at inference.
Key results:
- Across six fine-grained visual benchmarks (V*, ZoomBench, HR-Bench 4K/8K, MME-RealWorld EN/CN), VAD reaches Avg6 = 78.32 (4B) and 79.93 (9B), beating the best scale-matched baseline by 2.40 points (4B) and 2.80 points (9B), and beating Decomposed OPD specifically by 2.95/2.88 points.
- The 4B VAD checkpoint (78.32 Avg6) exceeds much larger closed/open models: Gemini 3 Flash (77.32), Gemini 3.1 Pro (78.04), Qwen3.5-397B-A17B (76.49).
- In a diagnostic, only 23.2% (4B) / 22.8% (9B) of the strongest teacher corrections were strongly aligned with evidence-conditioned response, showing most corrections are source-mixed.
- Ablation: replacing direct teacher matching with branch-aware VAD target reconstruction (no regularizer) raises Avg6 from 77.06 to 78.06 (+1.00), and adding the weak regularizer reaches 78.32.
- Held-out generalization (MMVP/CV-Bench/MMStar/POPE): VAD is the only post-trained method with positive delta vs. Base at both scales (+0.24 at 4B, +0.23 at 9B), while other baselines regress by up to -4.20.
Why it matters / caveats: Provides a training-time-only mechanism (no inference cost) for extracting the causally visual portion of teacher corrections; authors note it's limited to a single contrastive view pair per intervention and the projection gives semantic enrichment rather than a fully identifiable/clean visual-vs-nonvisual separation.
Progressive Agent Skill Generation via Reinforcement Learning →
Technical breakdown
Problem: Automatically generating high-quality, reusable "skill" modules for LLM agents from documents or execution experience is hard because skills lack a natural correctness/relevance supervision signal — their value can only be judged by how they change downstream agent behavior — and existing methods rely on heuristic, source-specific pipelines rather than a unified learned approach.
Method: Skill-α formulates skill generation as a sequential local-edit process over an action space A = {CREATE, UPDATE, MERGE, PRUNE, NOOP}, where a skill-editing policy πϕ (initialized from Qwen3-8B, SFT-warmed on DeepSeek-V4-Pro-synthesized edit trajectories) reads evidence units sequentially and edits the current skill state one step at a time. The key training signal is "rollback reward": for each candidate edit, the original and edited skills are each used to answer the same evidence-related anchored query with a fixed worker agent (GPT-4o), and a benchmark-specific verifier compares the two outcomes to assign a binary edit-level reward (with a NOOP fallback rule), which is optimized with GRPO (group size 8).
Key results:
- Under GPT-4o worker: on CL-Bench (document-to-skill), Skill-α reaches an average of 10.38 vs. the strongest baseline's 8.57 (No Skill) / 7.11 (best prompting baseline) — a +3.3 point gain over the strongest skill-generation baseline mentioned in the abstract, and Procedural Task Execution improves from 4.30 to 9.68.
- On experience-to-skill (SpreadsheetBench + tau2-bench, GPT-4o worker), Skill-α reaches tau2-bench average 55.83 vs. best baseline SkillPro's 49.17, a +6.7 point gain; SpreadsheetBench improves from 18.00 (No Skill) to 27.50, Airline from 40.00 to 65.00, Telecom from 12.50 to 22.50.
- Cross-worker transfer to Claude-Sonnet-4.5: Skill-α remains best or tied-best on all 4 CL-Bench categories (avg 9.55) and on tau2-bench average (70.33) despite skills being generated purely from GPT-4o trajectories.
- Ablation: full Skill-α outperforms "SFT only" by +6.92 (CL-Bench), +12.00 (SpreadsheetBench), +11.66 (tau2-bench avg); removing rollback reward drops performance close to SFT-only levels (3.68 CL-Bench avg vs. 10.38 full).
Why it matters / caveats: Rollback reward is shown to be the critical ingredient (its removal collapses gains), and the method generalizes across heterogeneous evidence sources (documents vs. experience) with one learned generator; a stated limitation is that the reward/verifier interface remains benchmark-dependent and the skill representation is text-only.
DAPD: Dual-Anchored Policy Distillation →
Technical breakdown
Problem: On-policy self-distillation (OPSD) conditions a teacher on privileged information (e.g., a reference solution) unavailable to the student at inference, causing "privilege illusion" — the student learns privilege-dependent behavior (e.g., asserting unsupported recalled answers) it cannot reproduce at test time, degrading performance as this information asymmetry compounds during training.
Method: DAPD combines Dual-Path Anchoring (DPA) and Dual-Source Anchoring (DSA). DPA introduces a "Self" distribution (student conditioned on the very completion it is predicting) as a trainable, information-matched bridge between the privilege-free "None" distribution and the privileged "Cross" distribution, forming an unconditioned path (aligning None↔None via Self, combining Entangled Distillation + an Inference Anchor loss) and a privileged path (aligning Self↔Cross via a Privileged Anchor loss). DSA then applies this DPA construction in both directions — Reference-to-Rollout and Rollout-to-Reference — combined with weight λ, so reliable-but-off-policy reference guidance and student-reachable-but-noisy rollout guidance are balanced; training uses LoRA (rank 64) on Qwen3 models with a component-clipped forward-KL divergence.
Key results:
- On Qwen3-4B across 6 benchmarks (AIME24/25, HMMT25, LiveCodeBench v5, BFCL v3, IFBench), DAPD reaches average 57.34 vs. OPSD's 55.34, a +2.00 point gain, and beats Purified OPSD (56.25) by +1.09 and DOPD (53.49) by +3.85.
- Gains persist across scale: OPSD's improvement over Base collapses from +5.19 (1.7B) to ≤+0.28 (8B–32B), while DAPD retains +2.41, +2.13, and +3.06 points at 8B, 14B, and 32B respectively.
- Behavioral probe: DAPD reduces late-stage "wrong claims" (unsupported answer assertions) by 73% relative to OPSD over training steps 250–300, and by 45% in the controlled Privileged-Anchor intervention experiment (with Avg@12 improving by +6.22 points in that intervention).
- Out-of-distribution transfer (reasoning-trained model tested on coding/instruct tasks): DAPD achieves best average 49.64, gaining +4.82 points on LCB v5 and +1.37 overall over OPSD.
- Ablation: combining both DSA sources reaches 65.28 Avg@12 (Qwen3-4B) vs. 63.89 (reference-only) and 65.09 (rollout-only); a reference-free "dual-rollout" variant still improves +2.41 points over OPSD at 4B/8B without any curated reference.
Why it matters / caveats: Identifies information asymmetry (not teacher quality) as the root cause of privilege illusion and shows this can be fixed by matching information availability rather than filtering/routing the privileged signal; the approach adds training-time compute (multiple anchored distributions per step) though no inference-time cost, and optimal loss-weighting coefficients must be recalibrated per model scale.
UEmbed: Unified Sparse and Dense Multimodal Embeddings →
Technical breakdown
Problem: Learned sparse retrieval (LSR) has remained tied to bidirectional encoder architectures (e.g., BERT/SPLADE) and text-only, auxiliary-module-based multimodal extensions, so there is no decoder-only backbone that natively produces both sparse and dense multimodal embeddings in one pass.
Method: UEmbed appends N=16 learnable special tokens to a decoder-only MLLM (built on Qwen3.5 at 2B/4B/9B scales) and partitions a compressed vocabulary (248,320 → 184,016 tokens via accent/case/whitespace merging) into N disjoint subsets via k-means clustering; each token's causal hidden state predicts sparse weights over its assigned subset via a linear projection with log(1+ReLU(·)) activation, concatenated into the full sparse vector, while the EOS hidden state preceding the special tokens gives the dense embedding. Training uses a unified loss combining dense InfoNCE (cosine) and sparse InfoNCE (inner product) plus FLOPS sparsity regularizers on query/document sides, with LoRA fine-tuning on attention/MLP projections and a frozen visual encoder, on 3.94M pairs from Echo-Embedding, MLDR, and MMEB with hard negatives mined via Qwen3-VL-Embedding-8B.
Key results:
- UEmbed-9B scores 71.8 (dense) and 71.0 (sparse) on MMEB-v2 (78 datasets), leading models trained on public data (vs. RzenEmbed-V2-7B 71.1, Ops-MM-Embed-7B 67.1).
- Sparse-dense gap stays within ≤1.0 point across all scales (e.g., 71.8 vs 71.0 at 9B).
- On BEIR (9 datasets, nDCG@10), UEmbed-9B dense reaches 56.3 average (best among compared models) and sparse reaches 55.2, matching specialist Echo-Mistral-SPLADE (55.2).
- Ablation vs. a matched bidirectional SPLADE baseline: +3.2 dense / +2.1 sparse points on MMEB-v1 image subset (61.3→64.5 dense, 61.3→63.4 sparse).
- On BrowseComp-Plus agentic search, UEmbed sparse mode needs fewer average search rounds (e.g., 31.05 for 9B sparse vs 33.68 for 9B dense) at comparable recall (64.33% vs 64.72%).
Why it matters / caveats: UEmbed shows sparse retrieval can be extended natively to multimodal decoder-only backbones without bidirectional conversion or auxiliary cross-modal modules, preserving vLLM/inverted-index compatibility. Limitations include English/Chinese-skewed sparse activations, occasional anomalous token artifacts from the large LLM vocabulary, and a larger dense-sparse performance gap in video compared to text/VisDoc.
WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity →
Technical breakdown
Problem: Existing world-model benchmarks mostly check whether explicitly specified controls or interaction outcomes are fulfilled, leaving underexamined a model's "inherent reactivity" — its ability to infer and generate scene-conditioned consequences (e.g., terrain adaptation, object contact response, social avoidance) that are not stated in the input.
Method: WorldExam is a hierarchical diagnostic benchmark with four levels — Visual Quality, Control Adherence, Spatial Consistency, World Reactivity — instantiated via eight tasks (Camera Control, Subject Control, Scene Revisit, Terrain/Object/Social Interaction, Physical Reaction, Goal Completion) over 1,474 cases, using "atomic control units" adapted to each model's native interface (SE(3) trajectories for camera-driven, discrete actions for action-driven, natural language for language-driven models). Evaluation combines geometry-based scoring (VGGT-Ω 3D reconstruction + SAM2 tracking for Camera Control, Scene Revisit, Subject Control, Terrain Interaction) with GPT-5.5 VLM-judge checklist scoring for the four remaining reactivity/goal tasks, plus an image-space displacement calibration procedure to normalize camera-driven translation inputs across models.
Key results:
- Benchmark: 1,474 cases, 8 tasks, 20 evaluated models (6 camera-driven, 7 action-driven, 7 language-driven).
- Camera Control: best camera-driven model NeoVerse scores 97.33 vs. best action-driven WorldPlay 92.74 vs. best language-driven Hailuo 2.3 63.29.
- World Reactivity gap: action-driven models achieve strong Subject Control (LingBot-World 55.47) but weak Terrain Interaction (best action-driven only 27.49) vs. language-driven Vidu Q3 reaching 64.39 on Terrain Interaction.
- Social Interaction: Veo 3.1 leads at 85.10 vs. best action-driven model at only 60.37; Goal Completion: HappyHorse 1.0 and Veo 3.1 reach 85.33/85.30 while Kling 2.5 scores only 48.25 despite strong visual quality.
- VLM-judge checklist agreement with human annotators: Spearman ρ=0.8614, PLCC=0.8583 across 5,793 checklist items/800 instances; reconstruction-backend robustness check (DA3 vs VGGT-Ω) shows only 3.09% mean absolute relative change in Overall score.
Why it matters / caveats: The benchmark reveals a clear capability split — no single model paradigm achieves both broad task coverage and strong reactivity, showing that visual quality and explicit control fulfillment do not guarantee "inherent reactivity." Scope is bounded by current model interfaces (dynamic-interaction track excludes camera-only models; Goal Completion is language-only), and metrics assess observable behavior rather than internal causal representations.
CADENA: Stepwise CAD Reverse Engineering →
Technical breakdown
Problem: Existing CAD reverse-engineering systems emit an entire parametric CAD program in a single pass without inspecting intermediate geometry, causing early errors to compound and conditioning generation on program text rather than the actual geometric residual still to be modeled.
Method: CADENA is a Qwen2-VL-based image-to-code VLM that grows a CadQuery/OpenCASCADE program one operation at a time, observing a single rendered multi-view image (6 orthographic + 2 isometric views, target in green channel, current partial build in red channel) plus a hint point locating the next edit region, then executes each predicted operation before deciding the next. Training uses two-stage supervised fine-tuning (1.86M short-program samples, then 18.0M samples up to 12 operations) from a rule-based procedural generator, followed by online RL with GRPO-style group-normalized advantages where the reward is volumetric IoU between the built and target mesh (candidates that fail to execute score zero), using one-step branching (32 candidates per step) rather than full-trajectory optimization. The paper also introduces CADENA-Bench (3,396 deduplicated real mechanical parts in 6 families) and the Generalized Match Score (GMS), a surface-type-matching metric (position + normal agreement) that works on non-watertight geometry.
Key results:
- CADENA-RL reaches GMS 97.0 on DeepCAD, 93.3 on Fusion360, 73.7 on MCB — best among all baselines (e.g., cadrille 94.8/86.8/55.0, CADEvolve 95.3/88.2/58.3).
- On CADENA-Bench (3,396 real parts), CADENA-RL scores 67.0 overall GMS vs. next-best CADFit 54.8 and CADEvolve 52.9; CADENA-RL leads 5 of 6 part families (e.g., shafts & bushings 73.0 vs. 59.6 next best).
- Invalid/build-failure rate on external datasets stays ≤1.2% for CADENA-RL, vs. 21.5–31.7% for CAD-Recode/CADReasoner on MCB.
- On BenchCAD's Vision2Code (mesh input, own selection protocol), CADENA-RL reaches 0.910 voxel IoU at 0.9% invalid rate vs. best frontier VLM (GPT-5.6 Sol) at 0.706 and best re-graded frontier entry at 0.355.
- Real mechanical parts are far harder than synthetic benchmarks: moving from DeepCAD to CADENA-Bench costs every method roughly half its GMS (e.g., cadrille 94.8→49.8).
Why it matters / caveats: Demonstrates that per-step geometric feedback plus IoU-based RL substantially improves reconstruction on real mechanical parts, but CADENA still trails CADFit's direct optimization approach on the gears & bearings family (58.1 vs. 61.3 GMS), and the BenchCAD comparison against frontier VLMs is explicitly not apples-to-apples since CADENA receives the ground-truth mesh as input and uses it for test-time selection.
SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation →
Technical breakdown
Problem: Simply providing LLM agents with Agent Skills (packaged instructions/tools) does not guarantee they can identify, apply, and coordinate them effectively, and there is no scalable pipeline for generating verified skill-grounded training data to teach this capability.
Method: SKT is a three-stage synthetic data pipeline: (1) skill curation via a rubric-based LLM filter selecting single- and multi-skill (k∈{1,2,3}) configurations with a composability judge; (2) template-driven task synthesis producing executable task packages passed through rule-based verification (file/path/evaluator checks, reference-solution execution, leakage detection), agent-based verification (semantic well-posedness plus paired with/without-skill rollouts to confirm skill dependence), and difficulty control (rejecting tasks with pass rate ≥0.6 over 5 rollouts), with feedback-guided repair looping back to task generation; (3) trajectory synthesis where a teacher model solves each task through an agent harness, with rule-based (full credit, well-formed tool trace) and LLM-based (skill consulted before use, applied correctly) trajectory validators retaining only successful, skill-faithful rollouts, used for masked autoregressive SFT on assistant tokens only.
Key results:
- From 2,000 public skills, SKT produces 4,000 accepted task packages (1,520 single-skill, 1,295 two-skill, 1,185 three-skill) and 27,164 verified trajectories (14,277 DeepAgents + 12,887 OpenCode) out of 32,000 candidate task–teacher–harness combinations.
- Matched-harness SFT on Qwen3.5-9B and Gemma 4 E4B-IT improves scores in all 16 model–harness–benchmark comparisons, gains ranging 3.20–18.91 points (e.g., Qwen3.5-9B+DeepAgents on SkillEval: 51.62→70.53).
- Skill-withheld condition shows only 0.53–5.69 point gains vs. 8.68–18.91 points when skills are provided, confirming the model learns to leverage external skills rather than internalize them.
- Unverified (no repair/verification) SFT data lowers all four benchmark means (e.g., -19.5 on AgentSkillOS-Bench) while SKT-verified data raises all four (+5.1 to +17.2), an 11.91–24.61 point gap.
- Cross-harness SFT transfer retains 49.1–58.1% of matched-harness gains; scaling training skills from 100→2,000 raises SkillEval score monotonically from 55.24 to 72.48 for Qwen3.5-9B–OpenCode.
Why it matters / caveats: Establishes verified synthetic data generation (not raw synthesis) as necessary for skill-use training gains, with benefits that generalize across harnesses and scale with skill coverage; a single mixed-harness checkpoint can nearly match harness-specialist performance (within 2.71 points), reducing the need for separate per-harness models.
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code →
Technical breakdown
Problem: Existing repository-level coding-agent benchmarks evaluate agents working alone or restrict user participation to chat messages, ignoring the common real-world scenario where users directly edit the shared codebase mid-task (found in 59.0% of sessions in the SWE-chat dataset), leaving agent robustness to such workspace changes unmeasured.
Method: SWE-Touch mines task-critical code regions by intersecting read/edit spans across multiple independent repair trajectories (from GPT-5.5, GLM-5.1, MiniMax M2.7) on the same task, then uses a separate "User Patch Generator" agent to construct a small, localized Counter-Edit near those regions — validated so the edit alone doesn't solve the task, the reference patch does, and edit+reference combined still fails. During evaluation, the runtime monitors agent actions and injects the patch (with a contextual user message) whenever the agent's action scope overlaps the edit region, up to K=3 times by default; evaluated with Mini-SWE-Agent shell interface across nine models on 200 SWE-bench Verified tasks (100-step budget) plus 25-task subsets of SWE-Bench Pro and DeepSWE (500-step budget, fixed-fraction delivery).
Key results:
- Counter-Edit lowers mean resolve rate by 7.7 percentage points across nine models on SWE-bench Verified, with per-model losses ranging 1.3–16.5 points and substantial rank reshuffling.
- Claude Opus 4.8 and GPT 5.5 are most robust (85.2%→83.3% and 80.5%→79.2%, retention 96.0%/95.0%), while Qwen3-Coder-480B drops most (57.2%→40.7%, -16.5 points, 60.8% retention).
- 63.3% of solved-to-unresolved failure trajectories terminate with the user's conflicting code still active (retained conflict); 13.9% show incorrect replacement, 11.6% incomplete reconciliation.
- Ablations: message-only intervention changes resolve rate by only -2.0 to +3.0 points, silent code-edit-only causes -1.0 to -9.5 points, and a non-conflicting Co-Edit control changes resolve rate by just -0.1 points on average (vs. -7.2 for Counter-Edit on the same 7 models) — isolating the effect to semantic conflict, not mere disruption.
- Revision rate (whether the agent removes/revises the user edit before termination) correlates with resolve-rate loss at Spearman ρ=0.80 across the 9 models.
Why it matters / caveats: Shows that strong autonomous coding benchmark performance does not ensure the state-awareness needed for shared-workspace collaboration; open-source models that score competitively on static leaderboards can degrade substantially under interactive conditions, indicating current optimization does not target this robustness. Findings point to detecting workspace changes, reconciling conflicts, and re-verifying affected behavior as key capabilities to develop.
Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs →
Technical breakdown
Problem: Autonomous-driving VLM chain-of-thought annotation pipelines that condition the teacher model on the ground-truth (GT) future trajectory induce "trajectory anchoring bias," causing the model to rationalize the revealed outcome rather than reason causally from scene evidence, producing less faithful and more hallucinated CoTs.
Method: The paper introduces AD-MCQ, which reformulates driving-trajectory planning as multiple-choice selection among K=8192 K-means-quantized trajectory-codebook candidates (built from 489K logged trajectories) with hard negatives selected by prototype similarity, avoiding open-ended coordinate generation. It then proposes DEFT-RLVR, a two-turn interaction trained with GRPO: Turn 1 elicits candidate-blind causal decision reasoning from scene context alone, Turn 2 reveals candidates only afterward for trajectory grounding; the reward combines exact-choice correctness (RMCQ) with an offline-generated, text-only VLM-judged rubric reward (RRUB) on the candidate-blind reasoning trace.
Key results:
- Human study: GT-conditioned CoTs showed 50.0% severe hallucination vs 29.0% for causal-planning CoTs, and only 24.0% pairwise win rate vs 60.5%.
- On Qwen3-VL-8B-Instruct, DEFT-RLVR raised AD-MCQ-500 accuracy from 28.1% (baseline JEFT) to 77.9%, and from 34.0% to 82.2% on Qwen3.5-4B.
- Under matched correctness-only reward, DEFT+RLVR beat JEFT+RLVR by 15.3 points (Qwen3-VL-8B) and 6.7 points (Qwen3.5-4B).
- General visual capability average improved from 54.81% to 56.09% (Qwen3-VL-8B) and 52.56% to 53.24% (Qwen3.5-4B), unlike SFT/distillation variants which dropped to ~49.8–51.8%.
- Cross-domain nuScenes transfer: DEFT-RLVR raised accuracy from 39.6% to 49.5% with CFS +0.114 and HLD +0.073.
- DEFT-RLVR added only 0.5% training-step overhead (426.5s vs 424.5s) over correctness-only RLVR while being 41.1% faster than an online-rubric variant (426.5s vs 724.4s).
Why it matters / caveats: The work exposes a general supervision-direction pitfall in CoT distillation for decision-making tasks (revealing the answer before reasoning), and offers a VLM-only, difficulty-controllable benchmark/training recipe that improves reasoning faithfulness without sacrificing (and slightly improving) general visual capability; findings are validated on only two VLM backbones and Waymo/nuScenes driving data.
WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning →
Technical breakdown
Problem: Existing critic models for VLA reinforcement learning estimate values from single-frame observations or weakly-supervised history embeddings, which is a fundamental mismatch with the partially observable (POMDP) nature of robot manipulation and yields poor value estimation.
Method: The paper proposes the World Critic Model (WCM), built on a lightweight LeJEPA architecture, with an observation encoder (ViT or the VLA's own VLM backbone) producing per-frame latents over a history window, a causal Transformer "world predictor" trunk, and two heads: a value decoder and a FiLM-conditioned action-dynamics decoder that predicts the next latent state. Training jointly optimizes a value regression loss (L2 to normalized return), a next-latent prediction loss, and a SIGReg isotropic-Gaussian regularizer to prevent representation collapse. WCM plugs into on-policy pipelines (PPO for OpenVLA-OFT, Flow-SDE for π0/π0.5) and off-policy pipelines (AWR, RECAP).
Key results:
- Across 149 tasks on ManiSkill, MetaWorld, CALVIN, and LIBERO-Plus with π0, π0.5, and OpenVLA-OFT backbones, WCM achieves SOTA IND/OOD performance, e.g. OOD avg. 51.5% (π0), 64.4% (π0.5), 77.9% (OpenVLA-OFT), each beating Flow-SDE/Flow-Noise/π-stepNFT/PPO/GRPO baselines.
- Zero-shot OpenVLA-OFT (0.78% success) improved to 98.7% after WCM-guided RL, a +97.9 point (~12,551%) absolute/relative gain.
- On LIBERO-Plus, starting from one-shot SFT (39.1% avg for π0), WCM-trained policies reached 72.8% after ~250 RL steps, exceeding full 50-shot SFT (71.2%).
- Real-world (WidowX-250S, 7 tasks): WCM (107.2M parameters) outperformed AWR and RECAP baselines on every task, e.g. stovetop cleaning 15/50 vs 10/50 (OpenVLA-OFT) and 33/50 vs 27/50 (π0.5).
- Ablation: replacing the world-prediction objective with λ=0 (ViT-only history) or MLP single-frame critics underperforms WCM; λ in [0.3, 0.5] gives best IND/OOD trade-off, and history length 3 performs best.
Why it matters / caveats: Demonstrates that explicit future-latent prediction (not just longer input history) is needed to make a critic's representation temporally informative, yielding large sample-efficiency and generalization gains in both simulation and real-robot RL; gains are largest from very weak SFT initializations and the method still trades off IND performance somewhat as λ is tuned toward OOD generalization.
Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representations →
Technical breakdown
Problem: Existing video motion-transfer methods rely on fixed structural correspondence (pose/skeleton, trajectories, or point tracks) between source and target, which breaks down when source and target objects differ substantially in morphology or motion mechanism (e.g., animal to inanimate object).
Method: The paper proposes a two-stage MMDiT-based diffusion framework. Stage I ("Abstract Motion Bootstrapping") grounds five complementary motion-view representations (ViTPose++ semantic kinematics, depth-aware global trajectory circles, TAPNext++ dense point tracks, SpaTrackerV2-derived 6-DoF rigid axes, and Canny edges) into a shared video-like conditioning interface via a flow-matching objective, then uses the trained model to synthesize cross-category video pairs that share motion but differ in appearance/morphology, filtered by four quality criteria (motion fidelity, target fidelity, leakage, video quality). Stage II ("Cross-Category Motion Internalization") reinitializes from Stage I and replaces the abstract-motion condition with the raw reference video itself, training the model to directly infer transferable dynamics without explicit motion extraction at inference.
Key results:
- New OpenVMT-Dataset (10K motion-equivalent cross-content pairs: 4K Same, 4K Near, 2K Far) and OpenVMT-Bench (123 I2V, 166 T2V test cases across Same/Near/Far splits), trained on ~1.6M Internet motion videos.
- On OpenVMT-I2V, the method achieves best HMF-All (0.7505 vs 0.7412 DisMo, 0.7349 Wan-Move, 0.7262 Tora) and highest Gemini motion score (3.575 vs ≤2.811 for baselines), with lowest leakage (1.000) and highest smoothness (0.9651).
- On OpenVMT-T2V, best HMF-All (0.7707 vs 0.7696 DisMo-T2V) and G-Mot. (3.976 vs ≤2.970).
- Human GSB preference: 93.0% overall preference (I2V vs DisMo) and 97.3% overall preference (T2V vs FlowMotion), with decisive win rates of 97.3% and 98.7% respectively.
- Ablation: replacing cross-category pairs with in-category pairs in Stage II drops HMF-All from 0.7707 to 0.7588 and G-Mot. from 3.976 to 3.436.
- Training: Stage I for 15K iterations, Stage II for 4K iterations, Adam batch size 128, 121-frame clips at 480×854.
Why it matters / caveats: Establishes a bootstrapping recipe to create cross-category motion-equivalent supervision without manually paired data, enabling a single model to transfer motion (e.g., kangaroo leap → lamp jump, jellyfish contraction → parachute) across large morphological gaps without test-time optimization or explicit motion extraction; evaluation relies partly on Gemini-based automatic scoring and a benchmark introduced by the same authors.
GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning →
Technical breakdown
Problem: Existing test-time latent-reasoning optimization methods (e.g., LatentSeek) connect latent states to the reasoning trajectory only through decoded tokens, creating an indirect, information-bottlenecked credit-assignment path that obscures how latent updates influence downstream reasoning and destabilizes optimization.
Method: GradCuit inserts optimizable latent states z(l) directly into an intermediate Transformer layer's hidden-state space (between prompt and continuation hidden representations, at l≈M/2 layers deep), so causal self-attention gives every generated continuation token a differentiable path to every latent through the remaining Transformer blocks and LM head. Latents are updated via policy-gradient-style ascent (z ← z + η∇z J) using reward-weighted gradients aggregated over the full continuation (Eq. 7), implemented with Adam (lr=1e-3) while keeping all base model parameters frozen; the latent count equals the token length of a fixed textual prefix ("Let's think about this problem and solve it step by step.").
Key results:
- Averaged over 5 backbones (LLaMA-3.2-3B/3.1-8B, Qwen2.5-7B/14B, Qwen3-4B), 3 benchmarks (GPQA-Diamond, GSM8K, MATH-500), and 2 answer formats (30 settings total), GradCuit reaches 64.5% average accuracy, beating CoT by 6.6 points and the strongest baseline (LatentSeek) by 2.4 points, winning 23/30 individual settings.
- Gains over LatentSeek by benchmark/format: +2.2/+2.0 pts (GPQA), +2.5/+3.8 pts (GSM8K), +2.8/+8.9 pts (MATH-500, Boxed/JSON).
- Learning-rate robustness: accuracy varies only 51.4–53.8% across 7 LR settings (std 0.82) vs 47.6–51.8% for LatentSeek (std 1.53).
- Random-walk (no reward guidance) variant still reaches 60.6% average accuracy, edging out reward-guided LatentSeek's 60.3%.
- Ablation (avg. over 15 backbone-benchmark settings): full GradCuit 66.6% vs fixed-prefix-only 62.0% vs plain CoT 60.5%; inserted prefix alone gives only +1.5 points, reward-guided gradient gives the decisive further gain.
- Gradient attribution shows reasoning-connector tokens (e.g., "because," "therefore") receive the highest gradient strength (up to 0.308 on GPQA-Diamond) across all three benchmarks; optimal latent-insertion depth is 25–50% of network layers.
Why it matters / caveats: Demonstrates that routing gradients through the model's own attention circuit (rather than through decoded-token log-probs) yields both higher accuracy and substantially more stable, interpretable test-time latent optimization without touching model parameters; results are confined to instruction-tuned models up to 14B and three reasoning benchmark types.
Roomer: Reflective Object-Grounded Model Editing and Repair for 3D Indoor Layout Synthesis →
Technical breakdown
Problem: Generated 3D indoor layouts can look globally plausible yet contain sparse local violations (collisions, out-of-bounds placement, blocked circulation, obstructed openings), and full-scene regeneration to fix them risks disrupting already-valid regions and introducing new violations.
Method: Roomer represents a layout as an object-addressable "RoState" and diagnoses violations into an object-grounded "RoReview" (issue type, implicated entities, relational roles, measurements). A geometry-conditioned vision-language planner (Qwen2.5-VL-7B-Instruct backbone with rank-8 LoRA and a geometry-token cross-attention adapter injected at decoder layers 22/24/26/28) proposes a schema-constrained "StatePatch" (action ∈ {MOVE, ROTATE, SCALE, INSERT, DELETE, REPLACE} + target + parameter seed); a deterministic solver validates the seed or searches an ordered fallback candidate set, and a candidate is committed only if full-scene re-verification confirms target resolution, no new hard violations, preserved protected relations, and structural validity. The planner is trained on Roomer-CC, a controlled-corruption dataset of 67,550 paired examples derived from valid 3D-FRONT layouts via six parameterized corruption operators with known-feasible inverse StatePatch targets.
Key results:
- On the frozen common-1100 benchmark (1,100 held-out 3D-FRONT/3D-FUTURE rooms), Roomer's final repaired layouts (Ours-Final) achieve best SCA Gap (12.46), OOB (8.66%), COL (17.50%), and Practical (82.98%) versus DiffuScene-RS, InstructScene, SemLayoutDiff-RS, and ReSpace baselines.
- Relative to its own initial layout, Roomer reduces OOB by 15.74 points and COL by 12.02 points, and raises Practical (usability) by 10.48 points.
- Transfer to 4 external generators: e.g. ReSpace OOB drops from 14.72% to 1.16% and Practical rises from 66.43% to 75.12% after repair; SemLayoutDiff-RS Practical rises from 57.77% to 66.86%.
- Ablation: removing RoReview (object-grounded diagnosis) collapses target resolution from 98.96% to 2.44% and Practical below the unrepaired baseline (50.39%); full-scene JSON rewrite causes 32.64% new hard-error rate vs 0.00% for Roomer.
- Adaptive stopping uses an average of 3.756 planner calls per scene, 62.44% fewer than a fixed 10-round budget.
- Professional validation (10 interior-design evaluators, 900 judgments): approval rate rises monotonically from 23.3% (Low Practical) to 56.7% (Medium) to 83.3% (High Practical), a statistically significant trend (Z=4.67, p=3.07×10⁻⁶).
Why it matters / caveats: Shows that verification-gated, object-grounded local repair can fix physical/usability defects in already-generated 3D layouts across multiple upstream generators without regenerating valid content, and introduces Roomer-Eval to measure "practical usability" (validated against professional judgment) beyond standard collision/FID metrics; the system is limited to its predefined residential rule set and finite action-specific candidate spaces, so issues outside this coverage remain unresolved.
GPTQ-2D: Cubic-Time Two-Sided Adaptive Rounding →
Technical breakdown
Problem: Extending GPTQ-style adaptive rounding to a two-sided (both left and right basis matrix) quadratic objective requires vectorizing the matrix, which naively costs quartic O(m²n²) time instead of the cubic time of standard one-sided GPTQ.
Method: The authors show that two-sided rounding under ∥A(Z−X)B∥²_F is equivalent to one-dimensional adaptive rounding of vec(X) under the Kronecker Gram matrix H⊗G, whose LDL feedback factor is separable as U^T⊗L; they prove that rounding errors propagate as rank-one updates confined to a lower-right rectangle, so entries on a shared anti-diagonal are independent and can be rounded in parallel. GPTQ-2D exploits this by maintaining a corrected matrix Y plus an auxiliary buffer C=LE and pushing each rounding error only along its own row and column (rather than densely over the whole rectangle), reproducing the exact same rounding trajectory as a dense vectorized sweep. A blocked variant (GPTQ-2D blocked) ports GPTQ's lazy block-update trick to anti-diagonals, replacing many small updates with a few band-like matrix products, and a padded "skew layout" is introduced so anti-diagonal, row, and column accesses all become fixed-stride array slices.
Key results:
- Reduces the two-sided rounding sweep from O(m²n²) (dense/vectorized) to O(mn·max(m,n)), matching one-sided GPTQ's asymptotic cost of O(m³) for m≥n.
- End-to-end complexity is O((max(m,n))³), cubic for square matrices, versus quartic for the naive dense realization — a factor of Θ(min(m,n)) speedup.
- Compared to YAQA's existing Kronecker-factored rounding implementation (O(mn·(max(m,n))²), quartic), GPTQ-2D can serve as a drop-in cubic-time replacement for its rounding step.
- Theorem 2 proves exact trajectory equivalence (bit-for-bit same rounded matrix Z) to the dense anti-diagonal and fully vectorized algorithms.
Why it matters / caveats: This is a theoretical/algorithmic result (no empirical LLM quantization benchmarks are run) that makes Kronecker-factored ("two-sided") weight rounding as cheap as standard one-sided GPTQ, which could benefit methods like YAQA that use Kronecker Hessian approximations for quantization. The paper explicitly does not address how to choose the basis matrices A and B (e.g., from Hessian estimation) and inherits GPTQ's lack of global-optimality guarantees, since exact closest lattice-point rounding is NP-hard.
DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents →
Technical breakdown
Problem: Existing multimodal deep-search agents and training data confine vision to the input or answer stage, so newly discovered images rarely drive subsequent retrieval steps, limiting interaction depth and reasoning span in long-horizon, cross-page/cross-modal search.
Method: The paper introduces EventVoyage-VL, a structure-before-language data synthesis pipeline that builds a multimodal event graph from Wikipedia/news (entities, relations, images with retrieval context and visual-content evidence), extracts subgraphs (visual co-occurrence networks, temporal event chains, spatial co-location structures, multi-relational motifs), composes them into inference programs requiring at least one visually-resolved internal interface, and verbalizes them into QA pairs with leakage/quality validators. DeepVoyager-VL is an agent framework with tools for candidate discovery (TextSearch, ImageSearch, ReverseImageSearch), goal-directed reading (WebVisit), active perception (FetchImage, CropImage), and computation (PythonInterpreter), decoupling image discoverability (lightweight references) from observability (explicit fetch/crop), with a helper VLM compressing tool outputs into goal-relevant summaries. Models (Qwen3-VL-8B/30B-A3B-Instruct) are trained via supervised fine-tuning alone (no RL) on curated teacher trajectories stratified by difficulty, for 4 epochs on 64 H20 GPUs, batch size 64, learning rate 2e-5.
Key results:
- DeepVoyager-VL-30B-A3B improves average score across 10 benchmarks from 20.7 (direct answer) → 40.7 (agentic workflow, +20.0) → 58.6 (trajectory SFT, +17.9 more); the 8B model goes 17.5 → 35.3 → 54.8.
- Achieves best reported result on 9/10 benchmarks (30B) and 8/10 (8B) among scale-matched open-source multimodal deep-search agents.
- Largest individual gains vs. its own agentic-workflow baseline: +22.6 on BrowseComp-VL, +23.7 on BrowseComp-V3, +30.7 on VisBrowse-Bench.
- With GPT-5 as backbone, DeepVoyager-VL's framework reaches 59.8 average across 4 benchmarks vs. 41.6 for LMM-Searcher (+18.2) and 22.2 for direct answer.
- Ablations: 20K multi-source trajectories raise average by 21.4pts (8B)/19.2pts (30B); adding 7K vision-in-the-loop (VIL) trajectories adds a further 5.4pts/6.5pts. Removing framework components (Summary, ImageSearch, FetchImage, CropImage) each drops average performance by 2.5–4.0 points.
- Visual tool calls make up 64.3% of EventVoyage-VL trajectories vs. 10.1–40.6% in three prior datasets.
Why it matters / caveats: Demonstrates that supervised fine-tuning alone (no RL) on data explicitly engineered to require intermediate visual dependencies can close much of the gap to proprietary models (30B model within 6.1–8.6 points of GPT-5.5/Gemini-3.1-Pro/Claude-Opus-4.6 under the same agentic workflow) while using an open 30B-scale backbone.
3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering →
Technical breakdown
Problem: Projection-based 3D vision-language models generate thousands of geometry-aware tokens per scene, causing heavy compute/memory overhead, and existing 2D-style or purely spatial token compression methods fail to address object-level token imbalance that persists even after spatial aggregation.
Method: 3DZip is a three-stage token compression pipeline built on the LLaVA-3D backbone: (1) coarse voxelization (voxel size δ=0.2m) that mean-pools tokens within fixed-size voxels to remove point-level redundancy from multi-view overlap; (2) feature-diversity-guided anchor selection using a Determinantal Point Process (DPP) with a cosine-similarity kernel over normalized voxel features, selecting K anchors via greedy Cholesky-based DPP maximization to capture semantically distinct objects; (3) spatially-constrained token merging that assigns each non-anchor token to its nearest anchor in feature space but only merges if the grid-space distance is within threshold τg=5, discarding tokens with no valid anchor.
Key results:
- Retains 94.7% of uncompressed LLaVA-3D performance (relative score) using only 128 tokens (vs. 1410 original), and 92.3% at 64 tokens, 88.9% at 32 tokens — best among all compared methods (FastV, SparseVLM, VisionZip, VisPruner, Voxelization, DTC) at every budget.
- At 64 tokens on SQA3D, improves EM from 50.2 (DTC) / 49.1 (VisionZip) to 52.8.
- On OpenEQA at 128 tokens, achieves 64.2 in attribute recognition (vs. uncompressed 64.0) and 53.2 in object recognition (+6.7 margin at 32-token budget).
- Reduces inference latency from 342ms/sample (LLaVA-3D) to 178ms/sample (48.0% reduction, 1.92× faster), FLOPs from 9.18T to 0.90T (−90.2%), and KV cache from 722MB to 101MB (−86.0%), with EM dropping only 2.5 points (55.7→53.2) on SQA3D.
- Ablation: feature-space distance beats spatial (XYZ) distance for DPP kernel (52.8 vs. 50.1 EM at 64 tokens); removing coarse voxelization drops 52.8→51.8 EM; removing spatial-constrained merging drops 52.8→52.3 EM.
Why it matters / caveats: Shows that feature-space diversity (not spatial proximity) is the key criterion for resolving object-level redundancy in multi-view 3D token compression, achieving a favorable accuracy-efficiency trade-off for deploying 3D VLMs. Limitations noted by the authors: coarse voxelization can attenuate small-object cues, and hyperparameters (δ, τg) are fixed across scenes rather than adapted to scene complexity.
DiffusionGemma Technical Report →
Technical breakdown
Problem: Autoregressive LLM decoding is fundamentally memory-bandwidth bound at low concurrency, and existing open-weight text diffusion models trade off between speed, reasoning capability, and multimodal support, with no open model being simultaneously fast, intelligent, and accessible.
Method: DiffusionGemma is obtained by fine-tuning the Gemma 4 26B-A4B mixture-of-experts model (3.8B activated / 25.2B total parameters, 8 of 128 experts + 1 shared expert active, 550M vision encoder, 262k vocabulary) into a discrete text diffusion model via a two-stage pipeline using under 10% of the AR model's training token budget: (1) supervised fine-tuning (SFT) that adapts the model to bidirectional attention over 256-token "canvases" using discrete multinomial diffusion with cross-entropy loss against ground-truth tokens; (2) an online "sampler distillation and reinforcement learning" (SD·RL) stage that jointly maximizes a reward objective and compresses the denoising trajectory via distillation. Inference uses block-autoregressive generation (causal encoder KV cache + bidirectional decoder per canvas) with an entropy-bounded sampler (max N=48 steps, entropy budget 0.1, adaptive-stopping threshold 0.005, temperature annealed 0.8→0.4) that accepts low-entropy tokens first and re-noises uncommitted positions.
Key results:
- Generates ~20 tokens per forward pass (TPF) on average, vs. 3–6 TPF achievable via state-of-the-art speculative decoding.
- Achieves ~1,500 output tokens/second on a single NVIDIA H100 GPU (FP8, batch size 1); third-party benchmarking reports up to 2,000 TPS on RTX 6000.
- SD·RL training yields a 10-point improvement on combined GPQA-Diamond + LiveCodeBench-v6 score and quadruples TPF from 5 to nearly 20 versus the SFT-only checkpoint.
- Adaptive stopping reduces average denoising steps to ~12 out of a max N=48 budget (a 4× latency reduction) without sacrificing quality.
- Each DiffusionGemma forward step processes 256× more tokens than an AR step but is only 3.2× slower in per-step GPU time (MoE kernel 4.3× slower due to ~84 activated experts per canvas vs. 8 for AR; sampling 3.06ms vs. 0.56ms; attention 4× slower).
- Final model uses under 5% of the total forward passes used by the Gemma 4 AR baseline across the eval suite, and post-SD·RL outputs are nearly 2× shorter than SFT-only outputs.
Why it matters / caveats: This is presented as the first openly-released text diffusion model that is simultaneously highly intelligent, exceptionally fast, and accessible (Apache 2.0 license), retaining thinking mode, multimodal input, long context, and even the ability to fall back to standard AR generation with only minor degradation. The report notes a performance penalty relative to the original AR Gemma 4 baseline as the cost of the diffusion adaptation, and factorized (conditionally independent) denoising steps can introduce local grammatical inconsistencies since simultaneously-updated tokens can't see each other's choices within a step.
LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation →
Technical breakdown
Problem: Talking-head video generation faces a latency-quality trade-off: multi-step diffusion methods are too slow for real-time/streaming use, while real-time autoregressive distilled methods suffer from error accumulation, identity drift, and loss of fine facial detail over long sequences.
Method: LeapTalk reformulates talking-head generation as data-to-data transport via a Brownian bridge (rather than noise-to-data flow matching), introducing "Bridge Forcing": each video chunk's trajectory is anchored to a persistent reference image I via Xt=(1−t)I+tX1+√(t(1−t))ε, with the first K frames of each chunk replaced by the last K frames of the previous chunk as a motion prefix for temporal continuity. To distill a pretrained flow-matching teacher (built on Wan2.1-T2V-1.3B DiT backbone) into a one-step Brownian-bridge student, the authors propose heterogeneous Distribution Matching Distillation (DMD) with an SNR-aligned time transformation Φ(τ)=1/(1+√((1−τ)/τ)) that aligns teacher and student noise levels for well-defined score matching, plus an audio-driven classifier-free guidance (CFG) mechanism (using Wav2Vec audio features) to preserve lip-sync under one-step generation. Two variants use different autoencoders: WanVAE (3D Conv, "Pro") and TAEHV (2D Conv, "Lite").
Key results:
- Achieves 1-step inference per chunk at up to 200 FPS (Lite) / 55 FPS (Pro) on a single H200 GPU, versus e.g. 0.15 FPS for FantasyTalking — up to 15,000× speedup, with per-video latency of 0.2–0.5s versus 1610–3542s for offline baselines (StableAvatar, EchoMimic, Hallo3).
- Best FID/FVD on HDTF (21/197) and CelebV-HQ (42/370) among all compared methods, and strongest lip-sync (Sync-C 8.38 on HDTF) despite using only 1 NFE (network function evaluation).
- Ablation: removing the Brownian Bridge formulation increases FID from 21 to 217 and drops Sync-C from 8.38 to 7.16; removing the SNR-aligned time transformation raises FID to 378; removing audio-driven CFG drops Sync-C to 4.34.
- User study (30 participants) shows LeapTalk preferred over baselines across identity consistency, lip-sync accuracy, visual quality, and overall preference in 91–98% of comparisons.
- TAEHV autoencoder cuts encoding/decoding time to 0.39s/0.24s (vs. 4.17s/5.26s for WanVAE) and memory to 0.008GB/0.411GB (vs. 8.495GB/10.128GB).
Why it matters / caveats: LeapTalk is presented as the first method enabling stable, open-ended, real-time streaming talking-head generation in a single diffusion step while preserving identity and lip-sync over long durations (demonstrated out to 720s+ in figures). The Lite variant trades some visual fidelity (slight blurriness in fine regions like lips) for large memory/speed gains, which the authors note can be mitigated by increasing to 2 inference steps.
StyleForge: Indoor Furniture Styling by Counterfactual Reasoning in a Hypergraph Field →
Technical breakdown
Problem: Given a fixed indoor furniture layout (categories, positions, orientations, scales prescribed) and a target style description, existing retrieval methods select assets independently or from static relations, producing scenes with color, material, and shape conflicts once composed.
Method: StyleForge maintains a learnable categorical candidate distribution per furniture slot, initialized from a frozen MLLM (Qwen3-VL) that extracts structured room- and slot-level style priors and a frozen Qwen3-VL-Embedding retriever that returns 15 category-compatible candidates per slot. A dynamic hypergraph style field propagates higher-order dependencies via style-conditioned activation of local and global hyperedges, and counterfactual style preference learning scores each candidate as a one-hot local substitution using scene- and candidate-level Mahalanobis energies with learned diagonal variances. Training alternates between a preference-ranking loss (Lrank + sparsity) that updates the style-field parameters θ and an energy-minimization objective that updates candidate logits α; at inference the MLLM and field stay frozen while only room-specific logits are updated via 200 steps of test-time training (TTT).
Key results:
- On 3D-FRONT (7,100 train / 800 test rooms), StyleForge reaches 79.1% Final R@1 vs. 44.5% for the strongest scene-aware baseline MetaFind (a 34.6-point improvement) and 33.9–36.3% for object-level baselines (ULIP, OpenShape, Uni3D).
- GPT-4o-judged scene metrics: StyleForge scores 4.53–4.64 (out of 5) across Aesthetic, Color&Material, Style Coherence, and Realism&Geometry, versus 3.04–3.23 for object-level baselines and 4.19–4.34 for MetaFind (GT is 4.58–4.67).
- TTT convergence: R@1 rises from 22.8% (step 0, initial retrieval) to 38.6% (step 20), 55.9% (step 50), 70.8% (step 100), and 79.1% (step 200).
- Professional validation: blinded interior-design evaluator approval rate rises monotonically from 22.2% (Low predicted AES) to 58.3% (Medium) to 86.1% (High).
- Ablation: removing the dynamic hypergraph, iterative refinement, or Mahalanobis energy drops Final R@1 from 79.1% to 61.7%, 23.7%, or 22.8% respectively.
Why it matters / caveats: The approach demonstrates that scene-level joint reasoning substantially outperforms per-object retrieval for style-consistent furnishing. A stated limitation is that StyleForge only reorders within a fixed initial top-K candidate set, so assets missed by the initial retriever can never be recovered.
DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents →
Technical breakdown
Problem: Predicting the future 6-DoF trajectory of a manipulated object from minimal input (a single RGB image and instruction) is hard because prior methods either need privileged inputs (video, depth, CAD models, initial pose) or must generate and perceptually parse a full video, which is costly and error-prone; existing egocentric datasets also lack fine-grained language-to-motion annotations.
Method: DreamTraj reads motion signals directly from a frozen Wan2.2-I2V-A14B image-to-video diffusion transformer at a single early denoising step (step 16 of 40, with stride-2 flow caching), extracting query-key attention tracks (a fixed block/head cell, block 28 head 32) as an implicit 2D point tracker and pooled hidden states (blocks 0/19/39, PCA'd to 128 dims) as scene geometry. A lightweight 7.1M-parameter flow-matching DiT "Reader" (4 blocks, AdaLN-Zero conditioning) denoises this 132-D readout jointly into 13 relative 9-D pose tokens (2D bearing + log-depth + 6D rotation) over 20 Euler steps, with metric scale restored only at output via a monocular/sensor depth estimate. Training uses conditional flow matching across two feature domains (real recorded clips and Wan2.2-generated clips supervised by their own extracted motion) to bridge the train/deploy feature gap, plus the newly curated Move dataset (5,038 human-inspected 6-DoF trajectories with fine-grained instructions, built via a selection→re-annotation→generative-augmentation pipeline).
Key results:
- Over all 5,038 out-of-fold samples (5-fold CV): 6.4 cm ADE, 9.5 cm FDE, 24.6° rotation error.
- Beats EgoScaler on its native window: DreamTraj ADE 3.04 cm vs. EgoScaler† 7.52 cm (real GT); rotation 7.9° vs. 28.6°.
- Beats ObjectForesight† (which uses privileged video context, mesh, and 3 ground-truth context poses) even on the extended whole-action horizon: ADE 6.76 cm vs. 10.29 cm, rotation 18.2° vs. 29.8°.
- Runs 76 s per 49-frame trajectory vs. 348 s for a generate-then-extract cascade — a 4.6× end-to-end speedup on identical hardware.
- Generalizes to 50 out-of-distribution scenes (real + video-game) with 80% success rate per 10 blinded annotators.
- Ablation: removing the q·k attention track costs the most ADE (+0.71 cm); removing pooled hidden states costs depth correlation most (0.640→0.565).
Why it matters / caveats: Demonstrates that dynamics knowledge can be "read" out of a frozen video diffusion model's internal latents rather than generated and perceived pixel-by-pixel, giving both accuracy gains and large efficiency gains for a minimal-input (image+instruction only) 6-DoF trajectory interface. The Reader is trained/evaluated on the newly built, relatively small (5,038-sample) Move dataset rather than an established large-scale benchmark.
Poplar: A Scalable Pipeline for Human-Centric Image Dataset Synthesis →
Technical breakdown
Problem: Producing individually convincing human-centric synthetic images does not guarantee a good dataset; existing pipelines lack mechanisms for collection-level semantic diversity, plausible attribute co-occurrence, everyday photographic realism, and auditable quality control at scale.
Method: Poplar is a three-stage Specify–Render–Inspect pipeline. Specify (Prompt Producer) samples structured attributes (identity, clothing, pose, scene, camera, style, etc.) under commonsense compatibility constraints C(a) = ⋀ck(a), then verbalizes them into photography-oriented prompts (separating a semantic layer from a capture layer describing photographer, device, framing) via Qwen3.5-27B-FP8 acting as a constrained verbalizer. Render (Image Producer) uses Krea 2 Turbo with the Krea2-realism-V2 adapter across five composition-aware aspect ratios (1:1, 2:3, 3:2, 3:4, 4:3) plus a deterministic preliminary visual filter (chroma/blur checks) that retries technical failures. Inspect (Quality Inspector) applies one Qwen3.5-27B-FP8 vision-language pass per image against a structured rubric checking intrinsic defects and prompt-mismatch severity, admitting only images with no intrinsic defect and no critical/major mismatch, while preserving the original prompt byte-for-byte.
Key results:
- Poplar-9K contains 9,401 curated image-text pairs retained from 11,765 reviewed candidates — a 79.9% acceptance rate.
- The dataset depicts 10,877 people across 9,401 images (some multi-subject).
- Cultural-background coverage across 13 sampled descriptors: individual groups range 6.9%–8.5% of images (relatively balanced).
- Clothing distribution: casual clothing most frequent at 18.8%, trendy feminine styles 15.5%, homewear 12.3%; most frequent scene category accounts for only 3.8% (long-tailed).
- Construction was run on 4 NVIDIA RTX 4090 GPUs; rendering used Krea 2 Turbo at scale 1.5, 8 inference steps, zero guidance, canvases ≤1,024px longer side.
Why it matters / caveats: Frames dataset construction as a distinct collection-level problem from single-image generation, with full provenance (prompts, hashes, attributes, inspection records) released for auditability. The paper explicitly notes Poplar-9K is small-scale/one instantiation, its demographic labels describe synthesis specifications (not verified identities), and automated VLM inspection remains fallible.
ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step →
Technical breakdown
Problem: Existing tool-use benchmarks expose semantic tool names/schemas in static environments, letting LLM agents rely on prior/memorized knowledge instead of testing true behavioral reasoning (inferring tool mechanics purely from interaction) or adaptation to non-stationary environments.
Method: ScrambleToolBench is an interactive terminal benchmark built on 28 core API tools and 20 procedural task templates, combined into 5-task episodes with a continuous shared context. It obfuscates all target command identifiers, parameter keys, output fields, and success/failure status tokens via a bijective mapping Φ (only meta-commands like submit_solution stay plaintext), forcing empirical discovery. Three environmental dynamics are added independently and combined: mapping drift (cyclic permutation of a fraction ρdrift of identifiers between tasks), stochastic action failure (probability pfail of a fake timeout on valid calls), and temporal execution windows (a k-step deadline after a trigger). The paper also proposes and formally analyzes a "cycle tracing" recovery algorithm and evaluates a memory-augmented agent baseline (persistent Task Recipes + Tool Knowledge databases).
Key results:
- Aggregate no-memory completion rate falls from 0.93 (unscrambled) to 0.32 (Base scrambled) to 0.03 under the combined +All condition (mean over 15 models).
- Claude Sonnet 5 and Gemini 3.1 Pro hit perfect 1.00 completion in unscrambled/Base, but Claude Sonnet 5 collapses to 0.00 under +All while Gemini 3.1 Pro reaches only 0.20 (0.50 with memory).
- Persistent memory recovers +0.09 completion rate and +0.59 tasks solved on average under +All (e.g., Gemini 3.1 Pro's +All completion rate rises 0.20→0.50, tasks solved 2.35→3.60).
- Optimal "cycle tracing" needs only k−1=6 extra calls per drift event (expected cost 4.25 actions via the derived formula), yet observed agent recovery costs are 5.57× (Gemini 3.1 Pro, high reasoning) to 8.94× (Claude Sonnet 5, low reasoning) that reference cost.
- Claude Sonnet 5's "follow the cheaper chain" rate is only 14.1% at high reasoning (vs. 10.6% random baseline, p=0.006); Gemini 3.1 Pro is statistically indistinguishable from random (11.0% vs 10.8%, p=0.934).
- Claude Sonnet 5 uses 7,326 completion tokens per solved task in Base and 11,652 under drift, 4.6× and 3.5× more than Gemini 3.1 Pro's 1,584 and 3,332.
Why it matters / caveats: Shows a fundamental agent reasoning gap: even top frontier models that build accurate internal tool maps default to expensive brute-force re-exploration rather than deducing structural changes (like permutation cycles) from their own stored knowledge, and scaling test-time reasoning amplifies search cost rather than enabling deduction. Evaluation is limited to 20 five-task episodes with a 100-action-per-task budget on a simulated (non-Docker) environment.
Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts →
Technical breakdown
Problem: Standard token-level MoE auxiliary balancing losses (Std-Aux) only observe the mixed image+text load, so opposing image and text expert-load errors can cancel out near one token ratio; when the image-text token mix shifts (due to resolution, image count, tiling, prompt length), the same trained router becomes badly imbalanced.
Method: The paper derives an exact "fixed-profile" law showing the mixed-load curve is a parabola in the image-token fraction a, with the image-text load-profile gap ‖qI−qT‖ controlling the curve's steepness/narrowness of its balanced region. Motivated by measured router-input geometry (strong modality separation; visual tokens clustering tightly by source image), it proposes ReBA (Relax Within, Balance Across): a Switch-style auxiliary loss with separate image and text terms (LReBA = λI·Linst(Mimg) + λT·Linst(Mtxt)), where the image term averages routing statistics per-image (one equal-weight routing instance per image, Mimg) rather than per-token, and text remains a single pooled instance. It's evaluated on four disjoint-neuron dense-to-MoE "split" backbones (Split-Qwen3VL-4B, Split-Qwen2.5VL-3B, Split-Qwen2VL-7B, Split-InternVL3-8B) after one epoch of SFT on Cambrian-737K.
Key results:
- On Split-Qwen3VL-4B, Std-Aux RMS-CV changes more than fivefold across five image-resolution settings, showing balance at one composition doesn't generalize.
- ReBA reduces mean-layer image-text correlation from −0.949 to −0.166 and mean ℓ1 conditional gap from 0.798 to 0.137.
- ReBA lowers mean layer CV on every benchmark and all four backbones (e.g., Split-Qwen3VL-4B average CV 0.69→0.43(Std-Aux)→0.16(ReBA)) while mean task accuracy stays comparable (differences of +0.3, +0.2, −0.2, +0.6 points vs. Std-Aux across the four backbones).
- ReBA lowers curvature κ of the composition-shift load curve by 96.3% versus Std-Aux; at the highest tested resolution, aggregate RMS-CV falls from 0.351 (Std-Aux) to 0.115 (ReBA).
- An idealized expert-compute proxy shows ReBA gives a 1.23–1.25× ideal speedup at medium/high visual loads.
- Ablation: image-only aggregation (Coupled-ImgInst) cuts image CV to 0.170 but text CV rises to 0.640 (overall CV only 0.097 due to residual cancellation); full ReBA achieves 0.099/0.110/0.077 (image/text/overall), the best in the table.
Why it matters / caveats: Provides both a theoretical explanation (the fixed-profile law) and a practical, near-zero-overhead (O(|M|N) per layer) fix for a load-balancing failure mode specific to multimodal MoE inference, validated across backbone sizes and under physical resolution/tiling shifts. The fixed-profile law explicitly excludes cases where physical preprocessing itself changes the conditional load profiles, and results are demonstrated on split (dense-to-MoE-converted) backbones rather than natively-trained large MoE models.
RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems →
Technical breakdown
Problem: Letting an LLM agent freely choose both what to modify and how to modify a recommender model leads to unstable, low-efficiency search under limited experiment (trial) budgets.
Method: RecHarness splits recommender optimization into a three-level control process: humans define the objective, validation metric, and a predefined set of edit arms (local arms for incremental refinement, jump arms for structural basin transitions); a Thompson-sampling bandit router maintains a Beta posterior per arm and allocates trials to the most promising edit directions using binary validation-improvement signals; and an LLM (conditioned on a dynamically updated textual "Experiment Skill" summarizing past successes/failures) generates concrete code mutations within the selected arm. A basin-aware jump mechanism activates structural-jump arms when the recent improvement rate over a rolling window falls below a threshold, with jumps accepted only if they beat the pre-jump incumbent by a margin after local retuning.
Key results:
- Improves average HR@10 by 85.85% over the GRU4Rec baseline and 12.58% over the stronger HSTU baseline on four Amazon Reviews datasets (sequential recommendation), under a fixed 43,200s GPU-time budget per model.
- On KuaiRec, reduces both watch-time/watch-ratio MAE by over 26% relative to the TPM baseline (e.g., WT-MAE 4.5372→3.3406, -26.37%).
- Ablations: RecHarness's non-baseline trials refresh the round-start best score 47.92% of the time vs. 22.45% (random routing) and 21.74% (LLM-selected arms); max single-trial gain is 24.00% vs. 10.16% for unrestricted LLM search without bandit structure.
- In a 7-day online A/B test on 10% of traffic in a short-video advertising ranking system, the RecHarness-discovered candidate (an intra-sequence self-attention module) improved ADVV by 2.084%, Revenue by 0.534%, and Exposure by 0.559% (offline production AUC +0.09%).
Why it matters / caveats: Demonstrates that separating "where to search" (bandit routing over structured edit dimensions) from "what to generate" (LLM reasoning) stabilizes and speeds up automated model iteration, with validated gains transferring from offline experiments to a live industrial ad-ranking system.
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures →
Technical breakdown
Problem: Outcome-level failure labels for LLM agents conflate distinct root causes (model, harness, environment, grader, etc.), making it unclear which component should be repaired — a "repair-assignment problem" that existing benchmark-specific taxonomies fail to resolve.
Method: The paper introduces an interaction-centric taxonomy that represents an agent system as interacting components (Model, Owner, Grader, Third party, Context, Memory, Tool, Local/External environment, and peer/subagent models), grouped into User, Harness, and Environment families; each failure is labeled as an interaction edge between two components plus a "fault side" (COMP1—COMP2 · fault: SIDE), using a root-cause attribution rule that traces backward from the observed failure to the earliest unrecovered event. The taxonomy comprises 41 failure modes grounded in worked examples from public benchmarks, system cards, and logged trajectories. Reproducibility is validated via an agent-as-a-judge pipeline (evidence reconstruction → classification → reflection/disambiguation) run with four frontier LLM judges (GPT-5.5, Claude Opus 4.6/4.7/4.8) against human-assigned labels on 40 worked examples.
Key results:
- Of the 41 failure modes, 36 are model-side and 5 are attributed to surrounding components.
- Best judge (GPT-5.5) reaches category-level agreement with human labels of Cohen's κ = 0.76 (accuracy 0.80, F1 0.69); complete failure-mode agreement is lower (accuracy 0.72, F1 0.64).
- Highest pairwise inter-judge agreement is κ = 0.84 (Claude Opus 4.6 vs. 4.8), comparable to judge-human agreement.
- Selective voting ensemble: requiring agreement among ≥3 of 4 judges gives 0.83 category precision at 90% coverage; requiring unanimity (4/4) raises precision to 0.96 at 68% coverage.
Why it matters / caveats: Gives a shared, reproducible vocabulary for deciding whether an agent failure calls for model post-training, harness engineering, or environment/evaluation redesign. Limitations: the taxonomy is descriptive (no frequency estimates), depends on evidence completeness in source material, and the agent-as-a-judge validation shows failure-mode-level accuracy remains limited (as low as 0.53 F1 for some judges), especially given ambiguous or incomplete source evidence.
ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures →
Technical breakdown
Problem: There is no large-scale, expert-curated benchmark for end-to-end multimodal comprehension and reasoning over domain-specific (ALD/etching materials-science) scientific figures, limiting evaluation of multimodal AI on authentic scientific chart understanding.
Method: The authors built Sci-ImageMiner, a dataset of 1,951 figures (49-category taxonomy) from 205 ALD/ALE research publications, extracted via MinerU into structured JSON+JPEG, pre-annotated with Qwen2.5-VL-7B-Instruct and refined by 10 domain experts (Fleiss' κ = 0.46 on a shared sample) via a custom annotation platform. The competition, run on CodaBench, defines four tasks: (1) Classification into 49 categories, (2) Data Extraction to Markdown tables (scored via RMS + TEDS), (3) Summarization (ROUGE-1/2/L + BERTScore-F1), and (4) Visual Question Answering across four answer types (Yes/No, Factoid, List, Paragraph), evaluated with task-specific metrics. Baselines used Gemma4 E4B, Qwen3-VL-8B-Instruct, GLM-4.6V-Flash, InternVL3.5-8B, and Molmo2-8B; top teams used Qwen2.5-VL-7B-Instruct/Qwen3.5-9B fine-tuned with LoRA/QLoRA, DPO/GRPO alignment, multi-agent inference, and RAG-based context injection.
Key results:
- Competition drew 68 active participants and 1,263 public/private submissions (Jan 9–Apr 8, 2026) across 4 task tracks (501+335+212+215 submissions).
- Task 1 (Classification): best team (Ricoh_SRCB) achieved F1 0.81 / Accuracy 0.79, vs. baseline Gemma4 E4B F1 0.68.
- Task 2 (Data Extraction): best team (TeleOCR-VL) achieved weighted score 41.81 (RMS 17.23, TEDS 66.39) vs. baseline weighted 35.97.
- Task 3 (Summarization): best team (DeepVitminC) achieved weighted score 0.56 (BERTScore-F1 0.84) vs. baseline 0.48.
- Task 4 (VQA): best team (DeepVitminC) achieved weighted score 0.31 vs. baseline (Molmo2-8b) 0.20; Factoid Exact Match remained low even for the top team (0.09).
Why it matters / caveats: Establishes the first expert-annotated end-to-end benchmark for ALD/E scientific figure understanding; results show current multimodal models handle classification/summarization reasonably but struggle substantially with data extraction and especially VQA/factoid reasoning, pointing to open challenges in domain-grounded scientific reasoning.
A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples →
Technical breakdown
Problem: Pixel-space diffusion models struggle to jointly capture global structure and high-frequency local texture, and existing fixes require training a new model from scratch rather than improving an already-pretrained model cheaply.
Method: The paper proposes Synthetic Self-Guidance (SSG): a lightweight adapter (1–2 transformer blocks) is attached to an intermediate layer (e.g., Layer 6 of JiT-B/16, Layer 8 of JiT-H) of a frozen pretrained pixel diffusion transformer (JiT, PixelREPA, DeCo), producing a coarse intermediate clean-image prediction; at sampling time the discrepancy between this intermediate prediction and the model's final prediction is used as a guidance direction, xguided = xinter + w(xfinal − xinter), analogous to classifier-free guidance but derived from the same frozen backbone. Critically, the adapter is trained not on real images but on the model's own generated (synthetic) samples using the standard flow-matching objective, while the backbone stays frozen throughout.
Key results:
- Adapter training costs less than 1% of full-model training compute (e.g., 63M adapter params vs. 953M backbone for JiT-H, 0.88% relative FLOPs).
- Without CFG, SSG reduces FID by more than 50% across evaluated JiT variants (e.g., JiT-H/16: 7.15→2.26; JiT-B/16: 25.42→9.47).
- With CFG, JiT-H/16 improves from FID 1.86 to 1.67 (10.2% reduction) and PixelREPA-H/16 from 1.81 to 1.59 (12.2% reduction); DeCo-XL/16 improves from 1.69 to 1.63.
- Synthetic-trained adapters outperform real-image-trained adapters in nearly all settings (e.g., JiT-H/16 w/o CFG: real-trained 2.51 vs. synthetic-trained 2.26); reducing synthetic dataset size 100x (1M→10K samples) changes FID by less than 0.05.
Why it matters / caveats: Offers a cheap, data-free (no real images needed) plug-in to substantially improve pretrained pixel diffusion models without backbone retraining. The authors note experiments are limited to class-conditional ImageNet generation with pixel-space models; generalization to text-to-image or other architectures/domains is untested.
GEOID-Flood: A Large-Scale Multi-Modal Benchmark Dataset for Flood Segmentation →
Technical breakdown
Problem: Existing flood-mapping datasets rarely combine bi-temporal SAR with co-registered optical imagery at scale or distinguish permanent water from transient flooding, leaving the value of geospatial foundation models for flood segmentation largely untested.
Method: GEOID-Flood is built from Copernicus Emergency Management Service (CEMS) Rapid Mapping activations, providing co-registered pre-/post-event Sentinel-1 (GRD and RTC, VV/VH), a pre-event Sentinel-2 composite, a Copernicus GLO-30 DEM, and a dedicated permanent-water layer derived from AlphaEarth Foundations (AEF) embeddings trained on the Earth Surface Water dataset. The construction pipeline (spatial partitioning, data retrieval, validity-mask generation, label composition, quality filtering) produces 3-class labels (background, permanent water, flooded water) with event-level, continent-stratified train/val/test splits (70/10/20%) to prevent spatial leakage. Benchmarking uses a shared U-Net decoder across geospatial foundation model encoders (TerraMind, DOFA, OlmoEarth, SSL4EO, Satlas) vs. ImageNet-pretrained encoders (ResNet, ConvNeXt, Swin), under three training scenarios: single-image, paired two-pass with flood-change loss, and paired single-pass early/mid fusion.
Key results:
- Dataset covers 219 flood events across 65 countries (2016–2026), 1,141,749 km² of flood-affected terrain, 14,282 tiles at 1024×1024 resolution — nearly double the area of the largest prior dataset.
- Best single-image model (finetuned TerraMind-L, 323M params) reaches binary IoU 0.884 / F1 0.936; Swin-T (32M params) reaches 0.873 IoU, within 0.01 of most finetuned foundation models.
- Best fusion configuration (finetuned early fusion, S2→S1 pre/post pairing) reaches binary F1 0.942, IoU 0.895, and flooded-water IoU 0.521, the best flood-class result reported (vs. 0.478 IoU_flood for the single-image post-hoc baseline).
- Cross-dataset generalization: training on GEOID-Flood and testing on a temporally disjoint held-out set (83 event-AoI pairs, 1,429 tiles from Feb–Mar 2026) yields binary IoU 0.845 (frozen)/0.854 (finetuned), outperforming training on MMFlood, Sen1Floods11, WorldFloods v2, or Kuro Siwo (best external: Kuro Siwo IoU 0.809/0.811).
Why it matters / caveats: Shows that training-protocol design (temporal pairing, optical-SAR fusion) matters more than backbone choice — foundation models offer only a modest, consistent edge over conventional encoders — and that GEOID-Flood's scale/diversity yields better transfer to unseen events than prior benchmarks. Limitations: geographic coverage skews toward Europe (140/219 events), labels inherit residual noise from CEMS delineations and the DL-derived permanent-water layer, and flooded water remains the hardest class to segment.