AI papers — 2026-05-18

Problem: Robot trajectories alone provide limited coverage for learning broad physical understanding, leaving a gap between visual-language model capabilities and embodied robot control.

Method: PhysBrain 1.0 builds a data engine that converts large-scale human egocentric video (Ego4D, BuildAI, EgoDex, EPIC, SEA-Small) into structured scene metadata—scene_elements, spatial_dynamics, and action_execution fields—augmented with depth estimates from Depth Anything v3. This metadata is converted into QA pairs across 25+ capability families using multiple annotator models (GPT-5, Gemini, Qwen variants) to train VLMs, which are then adapted to robot control policies via capability-preserving fine-tuning.

Key results:

• Single-object vegetable grasping success: 47.1% → 63.3% (Franka robot, 50 trials)
• Long-horizon semantic task success: 31.0% → 45.0%
• State-of-the-art results reported on ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa benchmarks

Why it matters / caveats: The work demonstrates that physical priors derived from human first-person video can effectively bootstrap robot policies with limited robot-specific data. Full quantitative breakdowns for most benchmarks are not provided in the technical report's accessible sections.

Problem: Visual agents lack reusable multimodal procedural knowledge, forcing them to rely entirely on model-internal priors without access to external step-by-step visual guidance.

Method: MMSkills packages procedural knowledge as state-conditioned skill units coupling a textual procedure, runtime state cards (when-to-use conditions and visible cues), and multi-view keyframes (full-frame, focus-crop, before, after). An agentic trajectory-to-skill Generator transforms public non-evaluation trajectories from OSWorld, macOSWorld, VAB-Minecraft, and Super Mario Bros through five phases—semantic clustering, skill proposal, merging, text-first drafting, and visual grounding. At inference, a branch-loaded agent isolates skill-environment alignment in a temporary branch and distills structured guidance back to the main agent.

Key results:

• OSWorld: Gemini 3.1 Pro 44.08% → 50.11%; Gemini 3 Flash 36.65% → 47.97%; Qwen3-VL-235B 21.34% → 39.17%; Qwen3-VL-8B 10.78% → 25.40%
• macOSWorld: Gemini 3 Flash 55.94% → 65.73%
• VAB-Minecraft: Gemini 3 Flash success rate 67.24% → 73.28%
• Super Mario Bros total score: 411 → 624 (Gemini 3 Flash)
• Trajectory length reduced: Qwen3-VL-235B from 15.22 to 9.87 steps; repetitive actions dropped from 21.8% to 6.2%

Why it matters / caveats: MMSkills shows that external visual procedural knowledge meaningfully complements model-internal priors across diverse agent tasks, even for frontier models. Limitations include dependence on source-trajectory coverage and inference overhead from branch loading.

Problem: Document QA systems are evaluated only on final answer correctness, ignoring whether models cite the correct evidence regions—a critical reliability gap for high-stakes domains.

Method: CiteVQA is a benchmark of 1,897 questions across 711 PDFs (seven domains, two languages, avg. 40.6 pages/document) requiring element-level bounding-box citations alongside each answer. An automated pipeline using MinerU2.5 parsing, MLLM-based evidence extraction, and ablation-based crucial evidence identification (masking individual document elements) generates ground truth. Evaluation uses Strict Attributed Accuracy (SAA), which credits a prediction only when both the answer and the cited bounding-box region are correct. Twenty MLLMs were evaluated.

Key results:

• Gemini-3.1-Pro-Preview: 76.0 SAA (best overall); 66.0 Recall, 86.1 Answer Correctness
• GPT-5.4: 59.0 SAA despite 87.1 answer correctness
• Best open-source model (Qwen3-VL-235B-A22B): 22.5 SAA
• Multi-document recall drops from 68.9% (single-doc) to 55.3% for top model
• Academic papers: 85.0 SAA; publishing/media: 63.3 SAA

Why it matters / caveats: The benchmark exposes "attribution hallucination"—models frequently produce correct answers while grounding them in wrong visual regions—a vulnerability that answer-only evaluations entirely miss. Results are currently limited to closed-source dominance; open-source models lag significantly.

Problem: Existing virtual try-on video methods are too slow for interactive use and cannot support garment switching mid-generation.

Method: FashionChameleon is built on an autoregressive video generation backbone fine-tuned with in-context teacher forcing using single reference-garment pairs (reference person wearing a different garment than the target). Streaming distillation via gradient-reweighted distribution matching improves long-video extrapolation consistency. A training-free KV cache rescheduling scheme with three operations—garment KV refresh, historical KV withdraw, and reference KV disentangle—enables interactive garment switching without retraining.

Key results:

• 720p generation at 23.8 FPS on a single GPU
• 30–180× faster than existing baselines (which achieve 0.13–0.77 FPS)
• HGC-Bench (240 samples): HGC score 4.6833, LGC 3.9250, temporal smoothness 0.9969
• User study: 672 responses confirmed superior preference for consistency and quality

Why it matters / caveats: First method to support interactive mid-generation garment switching in video at real-time rates. Evaluation is on the authors' own HGC-Bench dataset; independent benchmarks are needed to confirm generalization.

Problem: Existing dexterous manipulation benchmarks lack functionally grounded tasks that jointly test tool-use, bimanual coordination, long-horizon planning, and in-hand reasoning.

Method: DexJoCo provides 11 tasks implemented in MuJoCo, with 1,100 human demonstration trajectories collected via Rokoko Smartgloves + HTC Vive Trackers (~$2,300 hardware cost) and retargeted using GeoRT self-supervised retargeting. Five baseline policies are evaluated: ACT, Diffusion Policy (Transformer and CNN variants), π₀.₅, and GR00T N1.5, under two randomization regimes (rand-obj and rand-full).

Key results:

• Best single-task policy (π₀.₅, rand-obj): 52.5% ± 1.4% average success
• Under visual randomization (rand-full): π₀.₅ drops to 34.1% ± 2.9%; DP-T drops to 20.0% ± 1.4%
• Bimanual Assembly task: 0–5.3% success across all policies
• Multi-task training degrades performance: DP-T falls from ~50% to 33.2% ± 2.4%
• VLA (language-conditioned) model shows near-zero language following: NMI of 0.018

Why it matters / caveats: Reveals that current policies—including large pretrained VLA models—fail at fine-grained insertion, temporal memory, and bimanual coordination, motivating embodiment-aware representations and tactile sensing. All evaluations are simulation-only; real-robot transfer is not assessed.

Problem: Discrete image tokenizers used in autoregressive generation lose text legibility and facial detail through aggressive downsampling and vector quantization.

Method: InsightTok adds two localized perceptual losses to standard tokenizer training with a 16k codebook at 16× downsampling. The text perceptual loss detects text regions, extracts patches, and compares them in a pretrained text recognition network's feature space with area-proportional weighting. The face perceptual loss detects and landmark-aligns faces, then measures reconstruction using a face recognition network's intermediate features. Both specialized losses are combined with standard reconstruction, codebook, perceptual, and adversarial objectives.

Key results:

• Text reconstruction accuracy: +28.89 percentage points over second-best method
• Face similarity gain: +0.09 over IBQ baseline
• PSNR: 23.64 (competitive general quality)
• Generation: MagFace face quality 23.33 (highest), text rendering 95.83% normalized edit distance, GenEval 0.82, DPG-Bench 84.11
• Without area weighting: 0.51 rFID degradation

Why it matters / caveats: Demonstrates that targeted, region-specific supervision in tokenizer training—rather than architectural changes—suffices to close the text/face fidelity gap. Improvements are evaluated on a single-stage autoregressive generation pipeline; impact on larger or multi-stage models is not assessed.

Problem: It is not understood why on-policy distillation (OPD) trains large language models faster than reinforcement learning, limiting principled design of post-training methods.

Method: The authors analyze OPD through a parameter-dynamics lens on models from 1.5B to 32B parameters, showing it exhibits stronger low-rank gradient concentration (spectral-to-Frobenius norm ratio 36.8% vs. 32.7% for RL) and earlier alignment with the final update subspace. Based on these findings, they propose EffOPD, which triggers adaptive extrapolation at exponentially-spaced checkpoints (t=2^n), generating five candidate parameter steps along the estimated update direction and accepting them greedily via lightweight 50-sample validation.

Key results:

• EffOPD achieves >3× training speedup on mathematical reasoning tasks
• Early OPD checkpoint at 10% training progress recovers ~80% of final reasoning performance after norm scaling
• OPD effective rank: 2,341 vs. RL: 2,754; top-1% subspace norm ratio: 94.7% vs. 88.5%
• Validated across 7 benchmarks (Codeforces, Taco, AIME24/25/26, MINERVA, GPQA) at 4 model scales

Why it matters / caveats: Provides the first mechanistic explanation for OPD's efficiency, enabling a practical 3× acceleration with no extra trainable parameters. The extrapolation heuristic relies on local linearity of update directions, which may not hold for all task types.

Problem: Distilling long chain-of-thought reasoning from large reasoning models is bottlenecked by post-hoc trajectory selection, which lacks collaborative inter-teacher interaction during construction.

Method: CoRD enables K heterogeneous teacher LLMs (combined into the R1-Qwen-32B student setting) to collaboratively construct reasoning chains step-by-step. Teachers are prompted with explicit step markers ("<think> ### Step") for cross-model consistency. At each step, a meta-prover model scores extended trajectories by predictive perplexity against the ground-truth answer. Beam search with B parallel trajectories preserves diverse high-potential hypotheses, costing 49% of Monte Carlo Tree Search compute.

Key results:

• R1-Qwen-32B student trained on CoRD-Hetero: 79.6% on AIME24, 70.2% on AIME25 (surpassing best individual teacher: 78.9%, 67.9%)
• CoRD-Hetero vs. Curation baseline: 93.1% vs. 84.8% answer accuracy; 0.774 vs. 0.652 predictive perplexity
• MATH500: 94.8% (CoRD) vs. 93.4% (Curation); TaTQA: 95.2% vs. 88.2%; PubMedQA: 91.8% vs. 88.4%
• Beam search: 79.6% vs. 76.7% greedy vs. 75.8% MCTS on AIME24

Why it matters / caveats: Students trained on CoRD data match or exceed individual teacher performance, demonstrating that collaborative step-wise synthesis is superior to selecting complete traces. The method depends on a meta-prover model for perplexity scoring, adding inference overhead.

Problem: Applying GRPO to align video diffusion models is computationally prohibitive, typically requiring hundreds of GPU days per experiment for a 14B-parameter model.

Method: Flash-GRPO reduces GRPO training to a single denoising step via two mechanisms. Iso-temporal grouping enforces that all rollouts for a given prompt share the same timestep, factorizing advantage computation to isolate policy-induced variance from timestep variance. Temporal gradient rectification derives and normalizes a time-dependent scaling factor λ(t) that varies by orders of magnitude, applying the corrected loss ℒ_TGR(θ) = (1/G)∑A^i_t/λ(t)·r^i_t(θ). Both are applied to video diffusion models from 1.3B to 14B parameters.

Key results:

• 6× acceleration in training cost vs. Flow-GRPO
• VBench (1.3B model, 350 GPU hours): Aesthetic Quality 66.43 (highest), Subject Consistency 98.70 (highest), Object Class 90.00 (highest)
• HPSv3 reward: Flash-GRPO ~5.4 vs. full-trajectory Flow-GRPO ~5.1; Flow-GRPO-Fast suffers optimization collapse

Why it matters / caveats: Makes GRPO-based video alignment practical at scale by eliminating the timestep-confounded variance that caused instability in prior sliding-window approaches. Evaluation is primarily on VBench; real-world preference studies are limited.

Problem: RLVR training for LLM reasoning is limited by on-policy exploration collapse, where brute-force rollout scaling is computationally expensive and provides diminishing returns.

Method: NudgeRL uses Strategy Nudging to condition each rollout on a single sampled strategy-level context (e.g., a mathematical technique) appended to the prompt, with context dropout at rate p_drop=0.5 to prevent over-reliance. An Inter-Intra Group Advantage decomposes rewards into intra-context quality and inter-context reliability components controlled by λ=1.1. A distillation objective transfers context-discovered behaviors back to the base policy. Evaluated on Qwen3-4B-Instruct and Olmo3-7B-Instruct across AIME24/25, AMC23, MATH500, and Apex Shortlist.

Key results:

• Qwen3-4B: NudgeRL with 8 rollouts (avg. pass@1 0.489) surpasses GRPO with 32 rollouts (0.487)
• Olmo3-7B: NudgeRL 0.285 vs. best GRPO 0.281 at 32 rollouts
• Outperforms oracle-guided RL baseline (POPE) despite weaker supervision
• Consistently superior pass@k curves over GRPO-8 across full k range

Why it matters / caveats: Demonstrates that exploration quality via structured strategy hints matters more than rollout quantity, achieving 8× rollout efficiency. Limitation: context pools are generated offline and may become less informative as the policy improves.

Problem: Game world models treat NPCs as passive background pixels, lacking the ability to generate strategy-driven NPC behaviors that respond to player actions.

Method: ReactiveGWM modifies a diffusion-based world model trained on Street Fighter II and Street Fighter Alpha 3 gameplay videos (~10,000 annotated triplets per game). Player actions are injected via a lightweight additive bias broadcast across the DiT block patch grid. High-level NPC strategy (Offense, Control, Defense) is grounded through cross-attention modules trained on VLM-annotated behavioral categories. The player and NPC pathways are decoupled so the interaction module learns game-agnostic representations enabling zero-shot transfer.

Key results:

• NPC strategy following: 75.8% (SF2) and 79.8% (SF3) vs. vanilla baseline 43.4% and 41.8%
• Human evaluation: 86.0% strategy accuracy (SF2) and 77.2% (SF3) vs. 43.9% and 17.5% vanilla
• Zero-shot transfer accuracy: 64.6% (SF2) and 73.7% (SF3) without retraining
• Player movement accuracy maintained at 95–100%; attack accuracy 93.3–100%

Why it matters / caveats: First work to enable transferable, strategy-rich NPC control in video game world models without target-game annotation. Limited to two Street Fighter titles; generalization to non-fighting-game genres is untested.

Problem: GRPO's fixed token-level probability aggregation (arithmetic mean) cannot simultaneously concentrate gradients on high-importance tokens and bound gradient variance, limiting training adaptability.

Method: HölderPO replaces the arithmetic mean in GRPO's token aggregation with the Hölder mean parameterized by p: ρᵢ,ₚ(θ) = (1/|yᵢ| ∑ rᵢ,ₜ(θ)^p)^(1/p). Positive p concentrates gradients on high-ratio tokens; negative p bounds variance; p=0 recovers the geometric mean. A dynamic annealing schedule decays p from +2 to −2 across training to first amplify sparse signals then stabilize convergence. Evaluated on Qwen2.5-Math-7B for math reasoning and Qwen2.5-Instruct-1.5B for ALFWorld agentic tasks.

Key results:

• AIME24: 46.7% (p=3, exceeding prior ceiling of 43.3%)
• MATH500: 85.0% (p=−1)
• Average across 5 math benchmarks: 54.9% (7.2% relative gain over GRPO)
• ALFWorld: 93.8% success rate (28.8% relative gain over GRPO's 72.8%)
• Qwen3-4B-Base: +4.5 points over DAPO; Qwen3-8B-Base: +8.9 points over GRPO

Why it matters / caveats: Provides a theoretically grounded, drop-in generalization of GRPO that subsumes prior variants (geometric mean, clipped ratio) under one framework. Dynamic scheduling requires choosing an annealing schedule; sensitivity to this choice across diverse task types is not fully characterized.

Problem: LLMs still fail on hard competitive programming problems because they solve each problem in isolation without accumulating reusable strategies or learning from past failures.

Method: Solvita organizes four frozen-LLM agents—Planner, Solver, Oracle, and Hacker—each connected to a trainable graph-structured knowledge network updated by RL signals. The Solver uses patch-based repair (targeted edits vs. full regeneration). The Oracle generates certified test suites via four-stage validation. The Hacker launches semantic, stress-testing, and hash-collision adversarial attacks. Knowledge networks use contextual bandit policies with edge weights updated by execution verdicts and vulnerability discoveries. Evaluated on CodeContests, APPS, AetherCode, and live Codeforces with GPT-5.4, DeepSeek V4 Pro, and Claude Opus 4.6 backbones.

Key results:

• CodeContests pass@1 with GPT-5.4: 40.0% (single-pass baseline) → 82.42% (Solvita)
• Outperforms competing frameworks on 14/15 backbone-benchmark combinations
• Patch repair: 91.2% token savings vs. full regeneration; fewer iterations (3.74 vs. 5.18)
• All three Solvita variants reach Codeforces Legendary Grandmaster (≥3000 rating) within 12 rounds; bare backbones plateau at 2700–2850

Why it matters / caveats: Demonstrates near-doubling of competitive programming accuracy by coupling experience-driven knowledge networks with frozen LLMs, without modifying model weights. Cold-start requires ~5,000 training problems; patch repair can accumulate inconsistencies on globally flawed candidates.

Problem: Existing 3D scene datasets do not provide sparse, comparable, and geometry-consistent panoramic training data with complete coverage and low redundancy simultaneously.

Method: COVER (Coverage-Oriented Viewpoint curation with ERP Range-depth warping) is a training-free viewpoint curator that scores candidate equirectangular projection (ERP) viewpoints using a greedy two-pass low-resolution algorithm: it projects accumulated point cloud geometry into candidate ERP frames, classifies pixels as explained/new/conflicting, and computes st(v) = Gt(v) − 0.35·Lt(v). Applied to Blender indoor, HM3D, ScanNet++, TartanGround, and OB3D assets, building CM-EVS with 36,373 curated ERP frames across 1,275 indoor scenes.

Key results:

• Fixed-budget (K=4): COVER coverage 0.1032, conflict ratio 0.0170 vs. coverage-only baseline: 0.1055 coverage, 0.0193 conflict
• Stable plateau at λ ∈ [0.1, 0.5]; λ=0.35 yields 0.4127 coverage and 0.0175 conflict ratio (K=30)
• Cross-source consistency: 0.413 (Blender), 0.393 (HM3D), 0.735 (ScanNet++) at fixed λ
• ~4–7× fewer frames per scene than comparable baselines (median 25 vs. 100–168 frames)

Why it matters / caveats: Provides an auditable, geometry-consistent panoramic dataset construction method that dramatically reduces redundancy. Currently limited to static scenes; extension to dynamic environments is future work.

Problem: GUI agents that excel at region-tolerant clicking fail catastrophically at precision-sensitive geometric construction tasks (e.g., GeoGebra-style drawing) where actions must land on exact canvas coordinates.

Method: PAGER decomposes geometric GUI control into dependency-structured planning (constructing a task graph with valid geometric ordering) and pixel-level execution (predicting operation type, object type, and pixel coordinates conditioned on canvas state). Training uses pixel-grounded supervised fine-tuning to establish executable action grammar, followed by precision-aligned GRPO-style reinforcement learning with three reward components: operation-type matching, point-level coordinate accuracy, and geometric validity. The PAGE Bench evaluation set contains 4,906 problems and 224,497 pixel-level process-supervised actions.

Key results:

• PAGER task success: 23.78% vs. Gemini-3.1-Pro 5.82% (4.1× improvement)
• PAGER Overall Score: 29.52 vs. Gemini-3.1-Pro 24.36 vs. InfiGUI-R1-3B 10.66
• PAGER step success rate: 62.20% vs. best GUI specialist below 16.18%
• Human evaluation correlation: 0.9397 with expert judgment
• Full PAGER (29.52) vs. without parameter reward (20.07) and without action-type reward (24.52)

Why it matters / caveats: Exposes that strong semantic understanding (>88% action-type accuracy) does not translate to task success (<6%) in geometric GUI tasks—a capability gap hidden by conventional benchmarks. Currently scoped to planar GeoGebra-style construction; broader CAD and diagram editing are future work.

Problem: Vision-language models cannot perform dense 3D geometry prediction, requiring separate specialized models for depth estimation that forfeit multimodal reasoning capabilities.

Method: DepthVLM attaches a DPT-style depth head to a frozen VLM backbone, extracting features from three intermediate ViT layers plus LLM final hidden states at image-token positions, fused via RefineNet blocks. Training uses a two-stage schedule: Stage 1 trains only the depth head (frozen VLM) with scale-invariant log loss; Stage 2 fine-tunes end-to-end combining autoregressive language modeling and depth losses. Focal-length normalization removes dataset-specific biases. Evaluated on DepthVLM-Bench, a new unified indoor-outdoor metric depth benchmark across 9 datasets.

Key results:

• DepthVLM-4B: δ₁ accuracy 0.868 average across 9 datasets; DepthVLM-8B: 0.876
• Outperforms DepthLM-12B (0.730) and Youtu-VL-4B (0.603); matches DepthAnythingV3 (0.877)
• Inference latency: 0.42s for 256×192 vs. DepthLM's 13+ hours and Youtu-VL's 2.48s
• No degradation on MMB-EN, MMStar, and other VQA benchmarks

Why it matters / caveats: Achieves state-of-the-art metric depth with a lightweight head addition while preserving full multimodal capability, making a single model viable for both language and geometry tasks. Currently limited to depth estimation; broader 3D perception tasks (normals, 3D detection) are not addressed.

Problem: Image editing models fail on abstract multi-step instructions (e.g., "make this advertisement more vegetarian-friendly") because they lack structured planning coupled with execution feedback.

Method: The framework has two stages. In Stage 1, a multimodal LLM planner is fine-tuned via checklist-guided self-training: checklists specify what a satisfactory edit must accomplish, plans are generated conditioned on them, and the planner is fine-tuned by autoregressive likelihood on its own outputs (perplexity 4.89 vs. 61.25 for GPT-5 teacher imitation). In Stage 2, an orchestrator selects tools (including FLUX.1-Kontext and Qwen-Image-Edit) and spatial regions per sub-task, trained via policy optimization to maximize rewards from a VLM judge scoring instruction adherence, identity preservation, and visual quality. A lightweight verifier re-ranks top-k candidates at inference.

Key results:

• Instruction Following score: 4.196 (proposed) vs. 3.355 (Qwen-plan) vs. 2.32 (FLUX high-level)
• Trained orchestrator instruction satisfaction: 74.0% vs. untrained baseline 61.6%
• MagicBrush CLIP-T: 0.3256 vs. GenArtist 0.3067 and LayerCraft 0.3157
• GEdit Benchmark: Semantic Consistency 8.153, Perceptual Quality 8.030, Overall 7.604 (vs. Qwen-Image-Edit 7.560)

Why it matters / caveats: Shows that coupling on-policy planning with reward-driven tool selection outperforms both single-step generation and teacher-imitation pipelines for abstract editing. Evaluation is on advertisement images; generalization to other abstract editing domains is not confirmed.

Problem: 3D Gaussian head avatar reconstruction requires either hours of per-subject optimization or expensive preprocessing, limiting scalability to new identities.

Method: FFAvatar uses a Multi-View Query-Former that takes canonical FLAME mesh vertices as queries and fuses information across multiple unposed portrait images into a unified 3D Gaussian representation. An end-to-end FLAME estimator predicts expression, articulation, and head pose directly from pixels via photometric supervision, eliminating external preprocessing. Training follows three stages: scalable pretraining on monocular videos from 1M+ identities, multi-view fine-tuning on the Ava256 360-degree capture dataset, and optional identity-specific personalization converging in ~500 steps.

Key results:

• NeRSemble benchmark (1-view): PSNR 17.60, SSIM 0.75, LPIPS 0.23, CSIM 0.68 vs. LAM: 13.31 / 0.69 / 0.30 / 0.61
• FFAvatar 4-view: PSNR 18.78; + personalization: 25.78 PSNR, SSIM 0.94, CSIM 0.89
• +5.5 PSNR gain over prior SOTA LAM; +2.57 PSNR over GAGAvatar
• FLAME estimator: matches VHAP tracker (25.78 vs. 25.81 PSNR) at 200× faster speed (60 FPS vs. 0.3 FPS)
• Reconstruction: 2s (without personalization), 10s (with); animation at 49 FPS on single A100

Why it matters / caveats: Sets a new standard for feed-forward avatar reconstruction by combining large-scale monocular pretraining with multi-view fine-tuning, achieving personalization-level quality in seconds. Scalable pretraining on 1M identities was critical (removing it causes −8.36 PSNR); the approach requires at least one portrait image and may degrade for extreme poses.

Generated 2026-05-18 from HuggingFace Daily Papers API.