AI papers — 2026-06-16

Problem: Existing VLMs are turn-based by design and cannot proactively respond to events unfolding in a live video stream without being prompted.

Method: JoyAI-VL-Interaction is an 8B vision-first VL interaction model built on the JoyAI-VL 1.0 base. It frames interaction as a per-second decision (respond / stay silent / delegate) learned from time-aligned training data at scale. Video is encoded with AdaCodec, which adaptively spends fewer tokens on predictable frames to keep costs bounded over unbounded streams. A full deployable system wraps the model with pluggable ASR/TTS, long-horizon memory, a background-brain bridge (any API or agent), and vLLM serving — with two concurrent loops enabling asynchronous task offload while maintaining real-time user presence.

Key results:

• Preferred over Doubao's in-app video-call assistant in 77.6% of head-to-head comparisons across 6 everyday scenarios.
• Preferred over Gemini's in-app video-call assistant in 87.9% of comparisons.
• Wins 100% of comparisons on monitoring/alerting and never loses on real-time translation or counting — the tasks that turn-based systems structurally miss.
• Sub-second latency; supports continuous video over hours of streaming.

Why it matters / caveats: This is the first fully open, vision-driven interaction model released with training recipe, data, and a complete deployable system — a direct enabler for embodied AI, AI glasses, and companion applications. The model was trained on JD.com-internal data, and the weights/data release is scheduled for June 20 2026; real-world proactivity thresholds may need careful tuning per deployment context to avoid spurious interruptions.

Problem: Producing a high-quality data journalism article — from raw data to a verifiable, audience-ready multimodal story — currently takes a team of human reporters weeks, and no agent system handles the full pipeline end-to-end.

Method: Data2Story is a multi-agent "virtual newsroom" that orchestrates specialised roles (data analyst, writer, visual designer, fact-checker) under a shared task graph. Two innovations distinguish it from prior agents: (1) an Inspector module that links every claim, number, and visual asset to its source (data table, code execution, or external URL), making articles externally verifiable; and (2) multimodal generation that reasons about reader needs per topic — e.g., interactive maps for geographic pieces, audio clips for music stories — rather than defaulting to static text and charts.

Key results:

• Evaluated on 18 articles from diverse topics paired with expert-written published pieces.
• Human study with 53 participants across rubric dimensions: Data2Story scores strongly on transparency and data auditability.
• A code verifier re-executes every claim against the source data; claim-data alignment is competitive with human-authored articles.
• Human-authored articles retain a clear edge in editorial angle, creative design, and informative presentation.

Why it matters / caveats: The first end-to-end multi-agent framework for data journalism with verifiable provenance; useful for supporting human reporters in producing auditable data stories at scale. The human edge in editorial judgment and creative framing suggests the system is best positioned as a research assistant rather than a replacement.

Problem: VLA and video world-action models (WAMs) operate in 2D pixel or latent space, leaving 3D geometric cues (depth, scale, occlusion) implicit — limiting robustness to environmental changes such as camera viewpoint shifts.

Method: GAM (Geometric Action Model) repurposes a pretrained Geometric Foundation Model (GFM) as the full policy substrate. The GFM is split at an intermediate layer: shallow layers serve as an observation encoder; a causal transformer Future Predictor inserted at the split forecasts future latent feature tokens conditioned on language instruction, proprioceptive state, and action history; the remaining GFM blocks decode both future 3D geometry (depth head) and action chunks (action regression head) from the predicted future tokens in a single shared forward pass.

Key results:

• 55× faster inference than video WAM baselines.
• +9.7 percentage points improvement on camera-perturbation robustness benchmarks (LIBERO-Plus Perturbed Camera) over baselines.
• Matches or exceeds foundation-model-scale VLA/WAM success rates across simulation and real-world manipulation benchmarks while using substantially fewer trainable parameters.
• Improves real-world task success rate on in-distribution and perturbed camera settings simultaneously.

Why it matters / caveats: Demonstrates that jointly predicting future geometry and actions in a shared geometric latent space is both more efficient and more robust than 2D-based alternatives. The approach relies on a pretrained GFM being available, which currently limits applicability to domains with suitable GFM coverage.

Problem: Interactive world models must simultaneously achieve accurate camera control, long-horizon scene persistence across revisited regions, user-promptable events, and real-time inference across photorealistic, game-style, and stylized visual domains.

Method: DreamX-World 1.0 is initialized from Wan2.2 and trained through a progressive pipeline. E-PRoPE (an efficient variant of PRoPE) applies projective camera conditioning on spatially reduced tokens, cutting inference latency ~30% vs. PRoPE. Memory-Conditioned Scene Persistence retrieves earlier views via camera-geometry-based retrieval and uses residual recycling to tolerate imperfect memory latents. Event Instruction Tuning adds composable multi-entity event control. The bidirectional video model is converted to a few-step autoregressive generator via causal forcing + DMD-style distillation + long-student-rollout training. Post-distillation RL alignment (camera-control + video-quality rewards) recovers quality and controllability. Inference uses mixed-precision DiT, residual reuse, 75%-pruned VAE decoding, and async pipeline parallelism.

Key results:

• Camera-control score of 73.75, overall score of 84.76 on 5-second evaluation — outperforming HY-WorldPlay 1.5 (80.79) and LingBot-World (80.45) in overall score.
• E-PRoPE reduces inference latency by ~30% vs. PRoPE with comparable trajectory-following accuracy.
• Streaming at up to 16 FPS on 8× RTX 5090 GPUs.

Why it matters / caveats: The RL post-distillation alignment step is a broadly applicable recipe for recovering generation quality after aggressive diffusion-step reduction. Real-time performance currently requires 8× high-end GPUs (RTX 5090), limiting immediate deployment on consumer hardware.

Problem: In coding agents, repository exploration — locating relevant files and code regions — consumes a large share of the main agent's token budget and pollutes its context with irrelevant snippets, degrading both efficiency and task performance.

Method: FastContext decouples repository exploration into a dedicated subagent powered by specialised models (4B–30B parameters). These models are bootstrapped from reference-model (frontier LLM) exploration trajectories via imitation learning, then refined with task-grounded RL rewards across three skills: broad first-turn search, multi-turn evidence gathering, and precise citation generation. At inference, FastContext receives a natural-language issue description, issues parallel file-read, glob, and regex tool calls, and returns concise file paths with line ranges as focused context — not raw code — to the main agent. Integrated into Mini-SWE-Agent on SWE-bench Multilingual, SWE-bench Pro, and SWE-QA.

Key results:

• Improves end-to-end task resolution rate by up to +5.5% across SWE-bench Multilingual, SWE-bench Pro, and SWE-QA.
• Reduces main coding-agent token consumption by up to 60%.
• A 4B RL-trained FastContext model shifts the score–token Pareto frontier favourably vs. using the same frontier model for both exploration and solving.

Why it matters / caveats: Establishes that repository exploration is a separable, trainable skill — small specialist models handle it more efficiently than the same large model doing both roles. The gains depend on Mini-SWE-Agent integration; porting to other agent scaffolds may require re-training the reward model.

Problem: Whether frontier-level performance on verifiable reasoning benchmarks can be achieved within a strictly small-model (3B parameter) regime.

Method: VibeThinker-3B applies the Spectrum-to-Signal post-training paradigm in three stages: (1) curriculum-based supervised fine-tuning over progressively harder problems; (2) multi-domain reinforcement learning with verifiable rewards across math, coding, and logic; (3) offline self-distillation to consolidate learned reasoning patterns. At test time, Claim-Level Reliability Assessment (CLR) performs claim-level scaling by sampling multiple reasoning chains and scoring individual sub-claims rather than full solutions.

Key results:

• 94.3 on AIME 2026 (↑ to 97.1 with CLR test-time scaling).
• 80.2 Pass@1 on LiveCodeBench v6.
• 96.1% acceptance rate on recent unseen LeetCode contests (out-of-distribution generalization).
• 93.4 on IFEval (strict instruction following — no degradation from reasoning training).
• Matches or exceeds DeepSeek V3.2, GLM-5, and Gemini 3 Pro on AIME/HMMT benchmarks at 1× model size.

Why it matters / caveats: Motivates the Parametric Compression-Coverage Hypothesis — verifiable reasoning compresses into compact models, while broad general knowledge and long-tail competence require larger parameter counts. This positions small reasoning models as a complementary paradigm, not merely cheaper substitutes. The gains are specific to structured verifiable tasks; open-ended knowledge questions remain the domain of larger models.

Problem: Ensembling autoregressive language models by averaging next-token distributions does not transfer to Masked Diffusion Language Models (MDLMs), because MDLMs unmask tokens in arbitrary order — each model may operate on a different partially denoised sequence, making a shared "next token" undefined.

Method: TIE (Trajectory-based Iterative Ensembling) tracks the confidence dynamics of answer-relevant token positions across denoising steps within each MDLM. The key observation is that answer-related tokens exhibit stable, low-variance confidence trajectories when the model converges to a correct answer, and high-variance trajectories otherwise. At each relay interval, TIE identifies the model on the most reliable trajectory and transfers its partially denoised sequence to less confident models, which resume denoising from that intermediate state. Because leadership rotates across steps (different models contribute complementary strengths at different denoising stages), all participating models contribute to the final response.

Key results:

• TIE consistently improves over individual MDLMs across general reasoning, mathematics, coding, and planning benchmarks.
• Performance gains scale with constituent model quality; TIE is most effective when member models have comparable and strong individual capabilities.
• (Specific accuracy numbers are not stated for individual benchmarks in the paper excerpt.)

Why it matters / caveats: First practical ensembling framework for MDLMs; the confidence-trajectory diagnostic is a generalizable insight for understanding MDLM decoding quality. The relay mechanism introduces per-step communication overhead between models, so latency scales with ensemble size and number of relay steps.

Problem: Inverse rendering of outdoor urban scenes must recover geometry, material (albedo, roughness, metallic), and HDR lighting from sparse driving-video captures, while achieving both physical accuracy and visual photorealism — objectives that existing physically-based and generative approaches satisfy separately but not simultaneously.

Method: BRDFusion is a hybrid framework coupling a 3D Gaussian Splatting (3DGS) physically-based rendering (PBR) pipeline with a video diffusion model denoiser. The PBR module recovers explicit geometry, BRDF materials, and an HDR environment map via differentiable rendering following the rendering equation. A generative video model (DiffusionRenderer) is conditioned on noised PBR renders via SDEdit-style initialization, removing Monte Carlo artifacts while preserving physically correct shading from explicit lighting changes. During inverse optimization, the generative model acts as a learned regularizer to resolve material-lighting ambiguity. A sliding-window averaging strategy ensures temporal consistency across long driving captures.

Key results:

• Achieves state-of-the-art performance on novel-view relighting, night simulation (with local vehicle-headlight control), and dynamic object insertion with physically consistent shadows on real and synthetic urban scene benchmarks.
• Outperforms InvRGB+L (physically-based) and DiffusionRenderer (generative) on both photorealism and controllability metrics.
• (Specific PSNR/SSIM numbers are not included in the paper's extracted pages.)

Why it matters / caveats: Resolves the accuracy–realism tradeoff for urban driving simulation; explicit local light control (e.g., car headlights) is a critical capability not available in generative-only approaches. Evaluation is primarily on short urban clips; scaling to very long sequences may require more advanced memory management than the sliding-window strategy.

Problem: VLM-based video agents face three deployment gaps: (1) prohibitive API cost from processing dense streaming frames; (2) static agent scaffolds that cannot improve from failures; and (3) absence of workspace-style agentic benchmarks that require using video evidence inside executable environments.

Method: VisualClaw combines two mechanisms: (i) Hybrid encoding — a cascaded frame gate (perceptual hash → 128-dim CPU encoder → adaptive change gate) filters uninformative frames before VLM API calls; the skill bank uses hot/cold top-k injection so only the most relevant skills receive full-text expansion. (ii) Self-evolving skills and memories — correctly-solved examples accumulate in a memory bank; failures trigger an offline LLM evolver that uses relevant memories and failure cases to rewrite skill-bank entries without updating VLM weights. Per-skill utility tracking and pruning keeps the bank compact over time. A curated VisualClawArena benchmark (200 scenarios, 24.4 avg steps/scenario) tests agentic execution with tool use and executable checks.

Key results:

• −98% avg per-question API cost vs. full-frame upload (peak −99.3% on Video-MME Long).
• −25.9% cost vs. uniform-8-frame baseline under the same skill evolution.
• +3.85% avg accuracy improvement; peak +15.80% on EgoSchema with Gemini 3 Flash.
• VisualClawArena: +2.9% (Codex) / +3.2% (Claude Code Sonnet 4.6) macro accuracy over no-evolution baselines; −9.5% cost.
• A 1-hour streaming session reduced from ~3,600 API uploads to just 5–20 calls.

Why it matters / caveats: Demonstrates a practical path to always-on AI for edge hardware (AI glasses) by decoupling compute-intensive VLM calls from continuous video ingestion. Self-evolution without weight updates is a compelling personalization paradigm; the offline evolver latency and skill-bank quality degradation at scale remain open questions.

Problem: Multi-task recommender systems that use a Transformer encoder with separate task-specific MLP heads suffer from an information bottleneck (shared task-agnostic representation), gradient interference (the seesaw phenomenon), and an architectural mismatch between attention-based encoding and static feed-forward prediction.

Method: OneRank eliminates the encoder–predictor split by internalizing multi-task reasoning inside the Transformer stack. In the forward pass: task-specific tokens with mutual invisibility enable early bottom-up specialization; candidate-aware contextualization aggregates cross-candidate signals via situational descriptors; a dynamic matching-based scoring head replaces static MLP scorers. In the backward pass: cross-task gradient detachment isolates task-private parameter updates from shared knowledge extraction modules, preventing negative transfer. The result is task-conditioned information selection, candidate-aware contextualization, and controlled cross-task interaction — all within a single unified Transformer pass.

Key results:

• Significantly outperforms state-of-the-art MTL baselines (ESMM, MMoE, PLE, Transformer-centric variants) across multiple tasks on large-scale industrial datasets from Shopee.
• Maintains competitive computational efficiency compared to the encoder–predictor baseline.
• (Specific AUC/GAUC/NDCG numbers are not in the extracted abstract pages.)

Why it matters / caveats: Establishes a new architectural paradigm for industrial recommender systems by making the Transformer natively multi-task rather than using it as a task-agnostic backbone. Gains are measured on Shopee data; generalizability to other recommendation contexts (e.g., content, ads) warrants independent validation.

Problem: The adversarial robustness of autoregressive visual world models is unknown, and standard reference-based adversarial attacks cannot assess it because attackers lack ground-truth future videos or knowledge of subsequent user controls.

Method: BadWorld is a label-free adversarial framework for autoregressive VWMs. The self-supervised velocity attack (S1) disrupts the model's early denoising dynamics by attacking the predicted velocity space directly — using the model's own denoising dynamics as supervision plus a context-based history proxy, with no future video or action annotation required. Trajectory-adaptive bi-level optimization (S2) formulates a two-level attack: the outer loop mines hard control sequences (actions under which the current perturbation is weakest), and the inner loop updates the perturbation against those hard trajectories, producing a control-agnostic adversarial image robust to action-sequence variation.

Key results:

• Visually imperceptible perturbations (L∞-bounded) reliably trigger incomplete denoising, structural collapse, semantic drift, and control inconsistency in representative VWMs with both continuous camera control and discrete action control.
• The Velocity-Min objective consistently produces the strongest degradation across models and metrics.
• Trajectory-adaptive optimization further strengthens attacks across diverse hard trajectories.

Why it matters / caveats: Reveals fundamental temporal fragility in current visual world models, with direct safety implications for autonomous driving simulation and robot planning. The paper also highlights a practical privacy application: perturbations can prevent unauthorized interactive generation from a single image. The gap between lab-measured fragility and real-world deployability of the attack (e.g., in physical settings) remains to be quantified.

Problem: Embodied world models are fragmented across domains (manipulation, driving, navigation) and require domain-specific control interfaces; no single open model unifies them under a common action representation while maintaining physical plausibility.

Method: QWEN-ROBOTWORLD adopts a Double-Stream MMDiT backbone (60 layers) in which a frozen Qwen2.5-VL encoder provides the action understanding stream and a video-VAE provides the visual generation stream; the two interact via layer-wise joint attention throughout denoising. An action-language mapping framework converts 20+ robot embodiment types and 500+ action categories into a unified natural language action interface, yielding the EWK dataset (8.6M video-text pairs, 200M+ frames). Training follows a two-stage General + Expert Progressive Curriculum: T2I/T2V/TI2V pretraining on general data, then a 4-phase embodied SFT schedule (single-view → multi-view → multi-view concatenated → complex/cross-domain), with manipulation weighted at ~90% of embodied data.

Key results:

• 1st overall on EWMBench (4.60); motion fidelity (HSD) of 0.566 — +33% over the runner-up; scene consistency 0.914.
• 1st overall on DreamGen Bench (4.952) across three robotic embodiment subsets.
• 3rd overall on WorldModelBench (8.99), outperforming all open-source models; perfect physics adherence on Newton's laws, mass conservation, fluid dynamics, and gravity.
• Outperforms all open-source models on PBench (0.804); motion smoothness 0.990 (2nd among open-source).

Why it matters / caveats: The action-language mapping framework is a broadly applicable recipe for unifying heterogeneous embodied data; language as the universal action interface enables cross-domain co-training without interface redesign. The model still lags behind leading closed-source models on some metrics, and the natural-language action interface may lose precision for tasks requiring sub-millimetre control.

Problem: LLM agents accumulate verbose execution traces over long multi-turn sessions; existing text pruning and eviction approaches reduce token counts but break prompt prefix continuity, causing KV cache invalidation that negates the cost savings.

Method: TokenPilot introduces a dual-granularity context management framework. At the global level, Ingestion-Aware Compaction acts as an ingestion-gate harness that normalises and stabilises prompt prefixes and filters open-world environmental noise before content enters the agent's context. At the local level, Lifecycle-Aware Eviction tracks the residual task relevance of each context segment and schedules offloading on a conservative batch-turn cadence — evicting content only when its task relevance has fully expired, preserving prefix continuity for backend KV cache reuse.

Key results:

• 61% cost reduction (isolated) and 61% cost reduction (continuous) on PinchBench.
• 56% cost reduction (isolated) and 87% cost reduction (continuous) on Claw-Eval.
• Maintains competitive task performance vs. prior context management systems.

Why it matters / caveats: Identifies KV cache alignment as a first-class constraint for agent context management — a consideration absent from most prior text-compression work. The conservative eviction schedule means TokenPilot may be suboptimal in contexts where task relevance is hard to predict in advance.

Problem: Existing agent benchmarks evaluate code intelligence (e.g., SWE-Bench) or data-science intelligence (e.g., DS-1000) in isolation, but real-world software development typically requires both: navigating complex file systems and generating code that operates on large heterogeneous datasets simultaneously.

Method: CODA-BENCH provides a data-intensive Linux sandbox built on the Kaggle ecosystem. It contains 1,009 tasks across 31 thematic communities. Each task environment averages ~980 files (CSV, JSON, PDF, TEX, JPG, etc.), simulating realistic data scale and noise. Agents must discover relevant files through complex directory hierarchies and then generate code to perform data-driven analytical reasoning (e.g., data integration, statistical analysis, cross-format queries). All tasks have auto-verifiable ground-truth answers.

Key results:

• Top-performing agents achieve only a 61.1% success rate, even with state-of-the-art coding agents.
• Performance gaps widen on tasks requiring multi-type data integration (CSV + JSON + PDF).

Why it matters / caveats: Establishes that code + data integration is a qualitatively harder and more realistic evaluation frontier than either in isolation. The 38.9% failure gap on what are essentially "real engineering tasks" points to concrete agent capability gaps in file-system navigation, multi-modal file parsing, and data-code co-reasoning. Benchmark contamination is limited by using private Kaggle datasets not in standard training corpora.

Problem: Web agent benchmarks evaluate only terminal success, collapsing qualitatively different failure modes (navigation/exploration failures vs. execution/action failures) into a single binary score that provides no actionable diagnosis.

Method: WEBSTEP introduces 10 self-hosted websites and 1,800 task instances with controlled difficulty. Each website is implemented as a deterministic semantic MDP dual: the rendered GUI is generated directly from an underlying semantic state machine, so every GUI interaction maps exactly to a high-level semantic state transition that is automatically recorded without manual annotation. A process-level evaluation framework decomposes agent trajectories into exploration success, execution outcomes, and per-skill invocation patterns. Bifurcation analysis localises the single decisive step at which an unsuccessful trajectory first diverges from a successful one.

Key results:

• Three agents with near-identical terminal success rates (31–33%) diverge sharply in exploration-vs-execution profiles.
• On Housing, OpenAI CUA outperforms Qwen3.5 by +23.7% on commit actions but underperforms by −15.6% on filtering — opposite rankings hidden by the aggregate score.
• Performance differences between agents widen monotonically as task difficulty increases; easy tasks show near-identical success but hard tasks reveal sharp separation.

Why it matters / caveats: Converts web agent evaluation from a ranking tool into a diagnostic tool, enabling targeted skill improvement. The semantic MDP dual construction limits WEBSTEP to websites that can be implemented with deterministic state machines; real-world websites with dynamic or stochastic backends are not currently covered.

Problem: Adapting VLA robot policies to new tasks requires per-task teleoperated demonstrations (~18× slower to collect than human-hand video) and per-task fine-tuning (~24 GPU-hours per task for current WAMs), making deployment costs scale linearly with the number of tasks.

Method: RECAP (Retrieval-Conditioned Action Policy) trains a policy once on paired target-embodiment and cheap pool-embodiment (human-hand video) demonstrations, then freezes it. For new tasks, pool-side demonstrations are simply appended to a retrieval pool — no parameter updates. At each control step, the k nearest-neighbour retrieved trajectories from the pool condition the policy as a residual over Cosmos Policy action latents: retrieved trajectories supply coarse high-level motion and task progression; the frozen policy learns only the embodiment-specific correction needed to execute on the target robot. The WAM's future-image prediction objective enforces visual consistency between the retrieved trajectory and the predicted scene evolution, providing an alignment signal that is informative only when paired with retrieval on unseen tasks.

Key results:

• PushT (7 unseen goal angles): 34.9% vs. 6.0% for cross-embodiment baseline.
• RoboTwin 2.0 (5 unseen tasks): 31.5% vs. 26.0% baseline.
• 18× cheaper data collection (human-hand video vs. per-task teleoperation).
• Zero additional training for new tasks; pool-progression study confirms monotonic coverage growth.
• Validated on a real robot.

Why it matters / caveats: Establishes retrieval as a principled substitute for per-task fine-tuning in robot learning — a paradigm shift with major practical implications for deployment economics. The WAM's visual consistency objective is key to making retrieval effective, so the approach is especially well-suited to WAM-based policies (Cosmos Policy); gains on standard VLA architectures are smaller, suggesting the design is most valuable alongside future-image-predicting models.

Generated automatically from arxiv PDFs on 2026-06-16.