AI papers — 2026-08-21

Problem: LLM agent training environments are hand-built and static, so they cannot adapt to a specific agent's weaknesses or keep providing useful learning signal once the agent masters the existing tasks, while automated environment-generation alternatives are domain-specific and rely on costly, unreliable LLM-generated verifiers.

Method: The paper proposes EnvHarness, a programmable wrapper layer (analogous to an agent harness, but applied to the environment side) that reshapes a static environment's reset/step interface via three composable plug-in components — Stage (modifies the initial state through a sequence of state-manipulation actions), Contract (rewrites the action/observation/transition spaces via transformation maps), and Chain (composes two base environments into one extended episode) — without touching the environment's underlying logic or verifier. To automate configuring these components per policy, the authors introduce EnvRigger, a black-box loop with four stages (Observe, Diagnose, Write, Validate) that runs the target policy, diagnoses behavioral flaws from trajectories, synthesizes candidate components, and validates them via fresh rollouts before accepting/revising/rejecting them; downstream, skills are extracted from EnvHarness-generated trajectories using ReasoningBank, and RL experiments use GRPO with Qwen3-8B-base. Evaluation spans five benchmarks/four domains (ALFWorld, WebArena, SWE-bench Verified, OfficeQA, SpreadsheetBench) with policy/EnvRigger backbones Gemini-3.1-Flash-Lite (ALFWorld/WebArena) and Gemini-3.5-Flash (others), compared against baselines No Skills, Original Envs, GenEnv, VeriEnv, and SWE-smith.

Key results:

• Up to 9.0-point improvement over original environments on ALFWorld OOD (66.2→70.4 in-dist to OOD avg +5.9); WebArena avg +3.1 points; SWE-bench Verified SR +2.70 (47.67→52.58) with average steps reduced from 53.58 to 49.61 (9.8% fewer steps overall claim); OfficeQA EM +1.80, F1 +1.96; SpreadsheetBench Pass@1 +3.27, Mean Score +1.01.
• Outperforms domain-specific generators: beats GenEnv by 5.7 points average (8.5 points OOD) on ALFWorld; beats SWE-smith by 2.46 points SR while using 5.11 fewer steps per episode.
• RL (GRPO, Qwen3-8B-base): ALFWorld in-distribution SR 87.9 vs 81.4 (original envs); WebShop score 79.2 vs 75.6 and SR 67.4 vs 66.0; slight OOD dip on ALFWorld (88.8 vs 89.6).
• Chain component: reduces average steps from 53.58 to 41.96 (standalone SR 49.63, near baseline 49.88); combining Stage/Contract + Chain skills gives best results — SR 54.30, AS 43.12.
• Environment scaling on SWE-bench Verified (300 envs): EnvHarness climbs from 47.67 to 54.79 (+7.12 points) and keeps improving, vs. only 52.13 for original envs and 50.37 for generated envs at the same budget.
• Cross-model generalization on SWE-bench Verified (Gemini 3.1 Flash-Lite, Qwen3.6 27B, Gemini 3.5 Flash, Claude Sonnet 4.6): EnvHarness skills beat original-env skills by 2.7 to 3.7 absolute points across all four backbones; gains are largest for weaker models (+9.3 and +11.1 points for the two weakest vs. under 5.5 for the two strongest).

Why it matters / caveats: EnvHarness reframes environment construction as a domain-agnostic wrapping problem rather than per-domain authoring, preserving trusted original verifiers while letting the environment co-evolve with the policy — a scaling advantage since human-built and LLM-generated environments plateau under a fixed environment budget while EnvHarness keeps improving. The Chain component is excluded from the automated EnvRigger pipeline because EnvRigger cannot observe internal states of joined environments, and the approach still depends on the base environment supporting deterministic resets for validation.

Problem: Synthesizing high-quality terminal-agent tasks (instruction, environment, reference solution, executable verifier) is hard because multi-stage generation pipelines lose source information and let the four artifacts drift into mutual inconsistency, producing unsolvable or mis-evaluated tasks.

Method: FACET (Fine-grained Agentic Construction of Executable Tasks) is a three-stage pipeline: (1) Information Source Acquisition collects and filters skill packages from OpenClaw, ClawHub, and GitHub (yielding 71K+ valid skills), then extracts and judge-filters scenario–skill pairs into a scenario–skill repository; (2) Scenario Reconstruction and Reference Building uses five agentic modules (skill analysis, scenario exploration, association/filtering, evolution/recovery, information expansion) to build a five-dimensional scenario representation (goal, context, capability, state, I/O-tools), then generates aligned solution and instruction references (RS, RI) with a consistency-alignment check; (3) Executable-State-Grounded Task Construction builds/repairs a Docker-based environment (bounded to 3 repair iterations), then sequentially generates instruction → solution → verifier (Forward order) all grounded in the same realized container state, followed by validation and targeted per-artifact repair (up to 5 iterations) via a constrained router, packaged in the Harbor task format. Rollouts were generated with the Terminus-2 scaffold powered by DeepSeek-V4-Pro, and 1.2K successful trajectories from ~6K validated tasks were used for supervised fine-tuning of Qwen3.5-4B/9B/27B using LLaMA-Factory.

Key results:

• Dataset comparison (Table 1): FACET yields 6,078 validated tasks from 1.2K trajectories, with the highest average tests-per-task at 22.77 (vs. 16.60 Terminal-Lego, 6.18 Nemotron-Terminal, 5.51 Endless-Terminals, 3.98 TerminalWorld, 3.29 Tmax) and longest average trajectory length (11.86 turns) among compared datasets except TerminalWorld (11.94); FACET tasks show lower pass rates (P@1=27.00, P@3=35.00) reflecting stricter multi-check criteria.
• Terminal-Bench 2.1 fine-tuning gains: Qwen3.5-4B 17.60→24.72 (+7.12, +40.5% relative); Qwen3.5-9B 27.34→35.58 (+8.24, largest absolute gain); Qwen3.5-27B 40.82→47.57 (+6.75), coming within 1.49 points of the ~15× larger Qwen3.5-397B (49.06) under the same evaluation setting.
• Reference models for context: GPT-5.5 (xhigh) 78.00, DeepSeek-V4-Pro-Preview (high) 73.03, Claude Opus 4.7 (max) 66.10, Gemini 3 Pro (high) 65.80, Kimi-K2.6 59.93, GLM-5.1 (max) 58.70.
• Task analysis: 89.40% of individual verifier checks pass across teacher rollouts, but only 20.94% of completed rollouts achieve full task success; 54.00% of unsuccessful rollouts fail only one or two verifier checks.
• Generation-scheme ablation (100 scenario–skill pairs): Forward (I→S→V) achieves 46.5% initial validity vs. 24.2% (Reverse, I→V→S) and 37.5% (Joint, single-call); after repair, Forward reaches 83/100 final yield vs. 63/100 (Reverse) and 65/100 (Joint); paired comparison shows Forward beats Reverse on 29 vs. 9 pairs (p=0.0017, sign test), while Forward vs. Joint (27 vs. 18) is less conclusive (p=0.233).

Why it matters / caveats: The results support environment-grounded, sequential artifact generation as a key design principle for scalable, data-efficient terminal-agent supervision — a 27B model trained on only 1.2K trajectories nearly matches a model 15× larger. A caveat is that FACET's stricter, denser verification produces low absolute pass rates (P@1/P@3), meaning task "difficulty" partly reflects conjunctive multi-check scoring rather than pure task infeasibility.

Problem: Existing coding-agent benchmarks measure aggregate task success but give little insight into why agents fail when repairing real scientific software, where a defective patch can corrupt the scientific evidence underlying research conclusions.

Method: The authors introduce SWE-bench Science, a repository-level benchmark of 119 tasks drawn from 98 unique GitHub repositories across 20 scientific domains, built via a four-stage "Chain-of-Evidence Protocol" (source sampling/screening, snapshot freezing and reproduction, public-material abstraction with information isolation, and hidden-oracle/counter-calibration). Tasks are organized into three paradigms — Issue-driven, Expert-exploratory, and Engineering-integration — each with its own construction pipeline and hidden validators (anti-overfitting, mechanism-generalization, or end-to-end contract tests). Eight agent configurations are evaluated: GPT-5.6-sol with Codex (max), Claude-Opus-5 with Claude Code (max), Kimi-K3 with Kimi Code (max), GLM-5.2 with Codex (max), Nex N2 with Codex, DeepSeek-V4-flash-0731 with Claude Code (max), Qwen3.5-397B with Codex, and DeepSeek-V4-Pro-0813 with Claude Code (max). A paired ablation on a 91-task subset removes explicit scientific auxiliary information (rationales, upstream repairs, paper excerpts, expert guidance) while holding the code, environment, and executable context fixed, tested on GPT-5.6-sol (xhigh) and DeepSeek-V4-flash (high).

Key results:

• Benchmark: 119 tasks, 98 repos, 20 domains (Chemistry largest with 24 tasks, then Materials Science 16, Biology 13, Biomedical Engineering 12, Physics 11); 52 Issue-driven (43.7%), 49 Expert-exploratory (41.2%), 18 Engineering-integration (15.1%) tasks; input code averages 80,600 lines (range 174–2,029,051); reference patches average +117.81/-44.53 lines.
• Best overall Pass@1: Claude-Opus-5 (max) with Claude Code at 47.90% (Issue-driven 38.46%, Expert-exploratory 65.31%, Engineering-integration 27.78%), despite a 96.64% public score — all eight configurations score below 50% Pass@1.
• GPT-5.6-sol (max) leads private score (78.82%), Fail2Pass (72.30%), and Pass2Pass (97.66%); DeepSeek-V4-Pro (max) leads public score (100.00%) and Engineering-integration Pass@1 (44.44%).
• Weakest model, Qwen3.5-397B: 14.29% overall Pass@1, 51.79% private score.
• Failure-mechanism audit (Table 3): Claude-Opus-5 has the fewest total categorized scientific errors (58, plus 4 runtime/eval failures) and fewest misguided-exploration errors (2); DeepSeek-V4-Pro has fewest knowledge/abstraction errors (15) and fewest incomplete-repair/integration errors (19); DeepSeek-V4-flash has fewest generalization errors (6) but the most integration errors (48).
• Scientific-information ablation (91-task subset): for GPT-5.6-sol, adding scientific info raised public/private scores (96.70%→97.80%, 73.23%→74.06%) but lowered Pass@1 (36.26%→31.87%) while using fewer tokens; for DeepSeek-V4-flash, adding scientific info raised public score, private score, and Pass@1 (98.90%→100.00%, 61.21%→62.53%, 16.48%→23.08%) while using more tokens (input tokens 4.84M→7.40M).
• Task-level overlap: GPT-5.6-sol passed 29 tasks with info vs. 33 without (21 overlap, 8 only-with, 12 only-without); DeepSeek-V4-flash passed 21 with info vs. 15 without (12 overlap, 9 only-with, 3 only-without).

Why it matters / caveats: The results show scientific software repair remains far from solved (sub-50% Pass@1 even for frontier agents despite near-100% public test scores), and that supplying scientific domain knowledge is not uniformly helpful — it can aid weaker models but induce anchoring and reduce exact-repair success in stronger ones. The authors note limited task counts per domain and that their analysis of how scientific knowledge is actually used remains preliminary.

Problem: Existing camera-controlled video diffusion models can synthesize plausible novel-view videos from a monocular input but fail to maintain cross-view consistency when scaled to the tens of target views needed for high-fidelity 4D Gaussian Splatting (4DGS) reconstruction of a human, an issue the authors trace to the bounded attention context of a single DiT forward pass.

Method: 4DAnyone is built on Wan2.2-TI2V-5B (a 5B-parameter DiT video diffusion model) and adds two mechanisms: Reference Context Packing (RCP), which uses multi-scale patchify layers (kernel/stride (1,2r,2r), r∈{2,4}, inspired by FramePack) to compress the growing set of generated reference views into a fixed-length context (O(1) instead of O(N)); and Target Context Routing (TCR), which cyclically rotates 4-view target groupings at high-noise denoising steps to propagate global structure and fixes adjacent groups at low-noise steps for detail stabilization. Geometric guidance uses 3D-aware skeleton conditioning: a GVHMR-estimated 3D skeleton (40-keypoint subset of the 308-keypoint Goliath vocabulary) is rendered with depth-buffered (z-buffered) rasterization to resolve front-back pose ambiguity and injected via a zero-initialized skeleton encoder as a residual to noisy latents. Training uses a 3-stage curriculum (foreground-only DNA-Rendering → all multi-view data without masking → adding monocular data) with a flow-matching loss plus LPIPS perceptual loss (λ=0.25), on a newly built MVGameHuman dataset (38k videos, 24 cameras, 318 actors, in-house game engine) combined with SynCamVideo, DNA-Rendering, TedTalk, and Pexels; training runs at 704×1280 resolution, lr 1e-5, on 128 H20-3E GPUs (~3 days total across stages). Final multiview videos are lifted to 4D via FreeTimeGS reconstruction.

Key results:

• On DNA-Rendering / DyMVHumans (16 generated views), 4DAnyone outperforms MV-Performer, TrajectoryCrafter, and a fine-tuned ReCamMaster† across all three evaluation dimensions (generated-video consistency, generated-video reconstruction, 4DGS reconstruction), measured by PSNR/SSIM/LPIPS.
• 4DGS Reconstruction: Ours 24.15/0.863/0.159 (DNA-Rendering) and 23.28/0.846/0.117 (DyMVHumans), vs. best baseline ReCamMaster† 20.55/0.807/0.214 and 19.86/0.795/0.159 respectively.
• Generated Video Consistency: Ours 24.33/0.862/0.163 (DNA-Rendering) vs. ReCamMaster† 21.47/0.806/0.210.
• Generated Video Reconstruction: Ours 23.69/0.850/0.165 (DNA-Rendering) vs. ReCamMaster† 20.74/0.809/0.204.
• Ablation (Gen. Video Consistency, 8 DNA-Rendering sequences): w/o TCR & RCP: 21.09/0.766/0.216; w/o RCP: 22.03/0.780/0.203; w/o TCR: 22.21/0.788/0.196; Full model (Sliding routing): 22.63/0.796/0.191 (best), vs. Full (Random) 22.20/0.788/0.197 and Full (Strided) 22.06/0.786/0.198.
• Training data: MVGameHuman 38k videos/24 cams/318 actors; DNA-Rendering 51k videos/48 cams/548 actors; SynCamVideo 34k videos/10 cams/66 actors; TedTalk 42k monocular videos/413 actors; Pexels 20k monocular videos/1,411 actors.

Why it matters / caveats: The approach enables reconstruction-grade 4D human capture from a single casually shot phone video (no calibrated multi-camera rig needed), which could broaden access to volumetric/free-viewpoint video content creation; the authors explicitly flag deepfake, identity-privacy, and copyright risks and call for disclosure/consent when using it. DyMVHumans is out-of-distribution for all compared methods, and detailed failure-case analysis is deferred to the supplementary material (not captured in the excerpt reviewed).

Problem: Identity-preserving image generation becomes unreliable as the number of specified people in a scene grows, because models must bind each reference to a distinct person and location while existing embedding-based identity losses cannot reliably establish correspondence among many noisy predicted faces during training.

Method: WithEveryone is a unified multimodal model (transfusion-style mixture-of-transformers, text/reasoning predicted autoregressively, image latents via flow matching) that generates group images from five to ten reference identities. Each selected reference is injected as a 512-dimensional ArcFace embedding projected into an "ID token"; a structured Layout Chain-of-Thought (extending ATLAS-style planning) autoregressively predicts identity-layout bindings, face/body bounding boxes, and pose keypoints using a discretized 2,002-token coordinate vocabulary, which a deterministic renderer converts into a visual layout condition. Two novel training objectives are introduced: ID Representation Forcing (adapted from "representation forcing," Wang et al. 2026), which supervises a per-identity prediction aligned to ArcFace space before image synthesis, and the Layout-Grounded ID Loss (LG-ID Loss), a flow-compatible identity loss that reads face correspondence directly from layout annotations (cropping predicted/target images at annotated face regions) rather than relying on unstable Hungarian/embedding-based face matching. Training uses a 400K-sample group-image corpus built from real group photographs with automated face analysis, pose detection, and captioning pipelines; the joint loss combines next-token prediction, flow matching (λ=1.0), representation-forcing cosine loss (λ=1.0), and LG-ID loss (λ=0.5).

Key results:

• On a new identity-disjoint benchmark of 210 real group-image examples (5–10 references, stratified 60/50/40/30/20/10 by group size), WithEveryone achieves the highest Sim(Tgt) of 0.499 vs. 0.462 for GPT-Image 2, and Sim(Ref) of 0.540 (second only to GPT-Image 2's 0.583).
• Copy-Paste artifact score of 0.055 vs. 0.169 for GPT-Image 2 (lower is less "copied-face" behavior).
• Identity Coverage of 97.3% (0.973) with a Dup (duplicate) rate of only 2.8% (0.028), ahead of Seedream 5.0 Pro (0.913/0.065) and GPT-Image 2 (0.905/0.075).
• Highest CLIP-I (0.861, statistically indistinguishable from Nano Banana 2's 0.860) and competitive DINO-I (0.716); CLIP-T of 0.273 was not the highest.
• Scaling from 5 to 10 references, WithEveryone's ArcFace similarity drops only from 0.629 to 0.571, vs. GPT-Image 2 (0.593→0.496) and UMO (0.412→0.330).
• Ablations: LG-ID Loss alone raised Sim(Ref) from 0.339 to 0.506 and Sim(Tgt) from 0.304 to 0.435 (largest single contributor); the full model (P7) reaches Sim(Ref) 0.555, Sim(Tgt) 0.461, Count 0.869, Coverage 0.960; ID token + Representation Forcing gave smaller gains (Sim(Ref) 0.351→0.364, Sim(Tgt) 0.313→0.328).
• Plan IoU with the model's own plan reaches 0.773 (ground-truth layout: ~0.79–0.80), indicating plan execution converges well while plan prediction quality remains the larger residual error source.

Why it matters / caveats: The work shows that grounding identity supervision in explicit layout annotations (rather than embedding-based face matching) is what allows identity-preserving generation to scale beyond the 2–5 person range that prior open methods were limited to, matching or beating proprietary systems like GPT-Image 2 and Nano Banana on a held-out, identity-disjoint benchmark. The authors note the ground-truth-layout ablation (P3) is an oracle upper bound (not achievable at inference), and diagnostics point to plan prediction (not execution) as the main remaining bottleneck; CLIP-T text alignment is not best-in-class, and comparisons rely on the paper's own newly constructed benchmark.

Problem: Existing memory benchmarks for LLMs mainly evaluate whether information is correctly extracted, stored, and retrieved, largely overlooking how retrieved memories can distort a model's reasoning or beliefs and degrade its performance on the current task.

Method: The authors formalize "memory traps" (cases where response quality with memory is lower than without memory) and build MemTrapBench, a benchmark of 1,050 instances spanning four scenarios under two categories: Reasoning Fixation (Cognitive Bias, Trauma, Task Boundary) and Belief Distortion (Safety). Instances are constructed from manually designed seeds (Domain, Trap Mechanism, Ground Truth, Planted Prior), expanded via GPT-5.4 into 18–40 turn multi-turn dialogues through a three-stage adversarial process (Plant the Trap, Bury It in Noise, Spring the Trap), then filtered through a two-stage quality-control pipeline combining automated filtering and expert human review. They evaluate five memory strategies (FullText, LightMem, MemOS, SimpleMem, EverMemOS) and a no-memory baseline on Gemini-3-Flash-Preview and Qwen3-30B-A3B-Instruct-2507, using GPT-5.2 as the primary LLM judge (with Claude-Sonnet-4.6 for consistency checks) across four metrics: correctness, format, relevance, and efficiency. They then propose AdaptiveMem, a prompt-based, inference-time system-prompt skill that instructs the model to watch for four named risk types before using retrieved memory, requiring no architecture or parameter changes.

Key results:

• Without memory, Gemini-3-Flash-Preview scores 85.16% and Qwen3-30B-A3B-Instruct-2507 scores 81.83% average; every memory strategy reduces performance below these baselines.
• Best memory methods still drop >10 points: EverMemOS reaches 71.17% on Gemini (vs. 85.16% wo/Mem); LightMem reaches 70.13% on Qwen3-30B (vs. 81.83% wo/Mem). Weaker strategies (SimpleMem, MemOS, FullText) fall to 54.69%–60.67% (Gemini) and 62.87%–66.47% (Qwen3-30B).
• Cognitive Bias and Safety scenarios degrade most: Cognitive Bias scores fall to 46.66%–65.48% (Gemini) and 47.18%–56.64% (Qwen3-30B); Safety falls to 56.15%–69.70% (Gemini) and 56.15%–69.20% (Qwen3-30B).
• Ablation on Trauma trap: removing abusive feedback while keeping the same task/patient raises the average score from 69.43% to 84.33% (correctness 66.40%→91.07%); on Task Boundary, no-trap control reaches 94.39% (slightly above the 92.29% no-memory baseline) while the trap-inducing setting drops to 31.05%.
• Memory length ablation (Task Boundary subset): average score falls monotonically from 36.03% at 25% memory length to 31.05% at 100%, versus 92.29% with no memory; most of the added decline (3.40 points) occurs between 25% and 50% length.
• Evaluation reliability: GPT-5.2 average drops from 92.29% (wo/Mem) to 31.05% (Mem), a 61.24-point decrease; Claude-Sonnet-4.6 drops from 95.57% to 40.07%, a 55.50-point decrease — both judges agree on direction and magnitude.
• AdaptiveMem gains on MemTrapBench: +11.8 (FullText), +14.9 (LightMem), +11.3 (EverMemOS) points on Gemini-3-Flash-Preview; +4.2, +2.5, +2.6 points respectively on Qwen3-30B-A3B-Instruct-2507.
• On LongMemEval, AdaptiveMem improves 4 of 6 settings (up to +4.0 points on Gemini, +3.0 on Qwen) and leaves the other two unchanged, i.e., no degradation of standard memory performance.

Why it matters / caveats: The paper shows that adding memory can systematically hurt LLM task performance even when retrieved memories are factually correct and relevant, a failure mode current memory benchmarks don't capture; AdaptiveMem offers a lightweight, architecture-agnostic mitigation, but the benchmark itself is explicitly framed as an adversarially constructed stress test of harmful memory influence rather than a general measure of memory utility.

Problem: Agent Skills (portable modules encoding domain knowledge and procedures) are hand-authored or generated in a single LLM pass and lack a closed loop for improving from real interaction failures, because existing feedback comes from single-turn QA evaluation, which patches only first-exchange-visible gaps before the evolution gradient decays and stalls.

Method: SkillEvo is a closed-loop skill-evolution framework with two pillars: (1) trustworthy feedback generation, which recasts multi-turn user simulation from an evaluation endpoint into a feedback generator via an intent state machine (gating coverage), dual-sided orthogonal evaluation (separating simulator-side intent coverage from agent-side exposed-intent accuracy s_C), and collective attribution (classifying failures into Knowledge Gap, Capability Limit, or Evaluation Noise, with only Knowledge Gap driving revision); and (2) controllable skill governance, run by an independent Skill Governor that enforces fact consistency (a hard constraint checked against dual anchors S0 and S_{t-1}) and structural consistency (a soft constraint repairing knowledge bloat, reference breakage, and factual over-generalization in the Skill's directed-graph structure). The pipeline runs as Scenario Synthesizer → User Agent → Verifier → Collective Attribution → Skill Optimizer (bounded, evidence- and reference-anchored edits) → Skill Governor, iterated for multiple rounds. It is evaluated against Self-Reflection and a Single-turn QA baseline (representing SkillForge, Liu et al. 2026).

Key results:

• Evaluated on Tencent Cloud production data: 6 categories of cloud services, 9 production Skills, 98 skill-reference files, 2,000 tickets total.
• Per-round Overall TSR (%): Original Skill 30.0 (no updates); Self-Reflection 30.0→59.2→58.7→57.4→58.8 (oscillates); Single-turn QA 30.0→58.9→64.5→65.7→66.4; SkillEvo 30.0→59.4→71.3→77.9→81.8.
• SkillEvo surpasses Self-Reflection by 23.0 points and Single-turn QA by 15.4 points at final round; improves 51.8 points over the original Skills.
• Ablation: removing multi-turn interaction (variant a) drops TSR to 66.4 (matching the Single-turn QA baseline exactly, isolating the 15.4-point gain to the feedback source); removing governance (variant b) drops TSR to 78.6 (−3.2 points).
• Dual-sided evaluation: simulator intent coverage c_U = 98.9%; simulator fidelity (human-rated similarity, 200 sampled dialogues, 2 expert raters) = 95.3%; agent exposed-intent accuracy s_C = 71.1%.
• Governance effect: cross-round regression rate (RegR) declines from 28.2% (R1→2) to 21.1% (R3→4), a −7.1 point change. Knowledge bloat: +2.8% cumulative growth with governance vs. +16.2% without (nearly 6x larger).
• Verifier agreement with human expert consensus exceeds 90%.

Why it matters / caveats: The results support the paper's core claim that the ceiling on skill self-evolution is set by feedback quality (multi-turn vs. single-turn) and governance, not editing capacity or iteration count, and the system is deployed in Tencent Cloud production. Caveats: the production ticket dataset cannot be released (privacy/confidentiality constraints), and the framework requires Generator ≠ Evaluator model separation; human confirmation is still required before any revision reaches production.

Problem: Converting bidirectional action-conditioned video generators into causal, few-step world models is difficult because discrete keyboard states and continuous mouse motion must stay aligned with temporally compressed latent chunks throughout causal training and autoregressive rollout, and aggressive sampling compression otherwise couples visual, action, and cache errors across chunks.

Method: ForgeWM is a four-stage progressive framework built on a Matrix-Game 2.0-style action-conditioned image-to-video generator (with a dedicated ActionModule injecting frame-rate discrete actions via cross-attention and continuous mouse motion via windowed temporal attention). Stage 0 performs bidirectional domain adaptation with a flow-matching objective; Stage 1 performs teacher-forced causal training with block-wise causal attention over clean history; Stage 2 performs online causal consistency distillation (following Song et al.'s consistency models) using a frozen teacher, generator, and EMA copy over a 48-level noise grid; Stage 3 performs on-policy distribution matching distillation (DMD-style, per Yin et al.) over the student's own autoregressive self-rollout, using a frozen "real" denoiser (Stage 0 checkpoint) and a trainable "fake" denoiser. This produces budget-specialized students at steady-state denoising budgets of 1, 2, and 4 steps (ForgeWM-1/2/4). A "Replay-Time Refinement" deployment protocol re-noises and re-denoises a saved one-step draft offline with the same student at a larger schedule instead of regenerating from scratch. Training uses 40,000 clips from the GF-Minecraft dataset at 640×352 resolution, initialized from the public Matrix-Game 2.0 lineage; the recipe is also transferred unchanged to gamepad-controlled FPS gameplay (ForgeWM-CrossFPS) using the SCOPE cross-game FPS setting.

Key results:

• On Minecraft (Table 2, vs. Matrix-Game 2.0 and HY-WorldPlay), ForgeWM variants achieve the best reported values in 6 of 7 quality/control columns.
• ForgeWM-2: Imaging Quality 0.6865, LPIPS 0.6171, Flow Profile 0.9429, KCtrl 0.9740, Mouse Accuracy 0.8268.
• ForgeWM-1: lowest latency (168.2 ms) and highest throughput (72.10 FPS) vs. Matrix-Game 2.0 (370.9 ms, 32.35 FPS) and HY-WorldPlay (2164.3 ms, 7.54 FPS).
• ForgeWM-2/4 jointly rank first on KCtrl (action-sign accuracy); ForgeWM achieves the lowest reference LPIPS among compared systems.
• Human preference study (41 participants, 615 judgments, blind three-way): ForgeWM-4 preferred 68.8% for visual quality, 57.6% for action accuracy, 55.6% for spatiotemporal consistency, 60.7% pooled preference share vs. Matrix-Game 2.0 and HY-WorldPlay.
• Replay-Time Refinement (Table 3): Replay LPIPS 0.6155 vs. 0.6168 for direct ForgeWM-4 generation from noise (comparable reference quality), while Ddraft drops from 0.6187 (direct regeneration) to 0.1970 (replay), i.e., roughly 3x closer to the experienced trajectory.
• CrossFPS transfer: macro-average paired LPIPS of 0.656 across three games (Halo Infinite, Modern Warfare, a sci-fi shooter), with generated-to-reference motion ratio averaging 1.45.
• Test-time step scaling on frozen ForgeWM-1: Imaging Quality peaks at 2 steps, Subject Consistency at 4 steps, reference LPIPS improves through 4-8 steps then slightly regresses; Flow Profile and KCtrl remain stable, indicating extra steps mainly increase motion magnitude and latency rather than directional control.

Why it matters / caveats: ForgeWM demonstrates that a single progressive training recipe can yield multiple deployable latency/quality operating points (1/2/4-step) while preserving fine-grained keyboard-and-mouse control fidelity, and that offline replay refinement can improve visual quality without discarding the user's experienced trajectory or requiring a second checkpoint. The quantitative comparison is intentionally restricted to a controlled Minecraft setting (matched initial states and control traces) because HY-WorldPlay uses a different control parameterization requiring a deterministic adapter; broader out-of-distribution generalization and long-horizon degradation (e.g., block-structure loss, spreading color artifacts at later timestamps) are noted as open limitations.

Problem: Existing code-generation agents assume a repository architecture is either predefined or produced once via a fixed, one-shot planning graph, which breaks down in "zero-to-all" generation where an agent must build an entire modular software project from scratch from natural-language requirements alone.

Method: Repo0 maintains a persistent architectural state, a Dual-Directed-Acyclic-Graph (Dual-DAG), composed of a requirement-level DAG (functional coordination between requirement units), a component-level DAG (implementation dependencies), and an alignment relation linking requirements to components. Starting from a requirement document, it builds an initial architecture via LLM-driven requirement extraction/merging and a three-stage reasoning-then-labeling decomposition (inspired by Atom of Thoughts), then iteratively evolves component boundaries using four structural actions (split, merge, revise, save) triggered by two modularity metrics (cohesion, coupling, defined via graph partitioning/minimum-cut and Jaccard similarity) plus a connectivity criterion, until structural convergence. After convergence, code is generated with a Test-Driven Development workflow (skeleton → tests → implementation → validation/repair). Backbones tested: GPT-5 mini and DeepSeek V3.2, evaluated on the RepoCraft benchmark (six real Python repos: scikit-learn, pandas, sympy, statsmodels, requests, django, paraphrased as MLKit-Py, TableKit, SymbolicMath, StatModeler, HttpEasy, PyWebEngine) against baselines mini-SWE-agent, Paper2Code, and RPG (state-of-the-art static Repository Planning Graph method).

Key results:

• Across all six repositories and both backbone models, Repo0 achieves the highest Functionality Coverage and Pass Rate in every setting.
• Vs. RPG (strongest baseline): Functionality Coverage improved by 4.55–20.08 percentage points; Pass Rate improved by 7.61–29.74 percentage points.
• On requests/statsmodels/django (GPT-5 mini): Repo0 Coverage = 100.00% / 80.68% / 80.50% (vs. RPG 90.91% / 70.40% / 60.42%); Pass Rate = 50.98% / 85.51% / 74.36% (vs. RPG 31.51% / 77.90% / 47.33%).
• On the same repos under DeepSeek V3.2: Repo0 Coverage = 100.00% / 78.41% / 79.17%; Pass Rate = 78.08% / 69.03% / 74.07%, all highest among methods.
• Ablation (RQ2): removing Structural Evolution caused the largest drops (e.g., requests Coverage −5.68, Pass Rate −8.47, Voting Rate −17.86; django Coverage −5.92, Pass Rate −13.33, Voting Rate −8.34); removing the Dual-DAG, Requirement Context, or Component-Graph Ordering each also degraded performance (e.g., statsmodels Pass Rate −30.00 without dependency-aware generation order).
• RQ3: on statsmodels with GPT-5 mini, metrics-guided evolution improved Coverage from 75.90% to 80.68%, Novelty from 11.05% to 11.48%, Pass Rate from 81.90% to 85.51%, and Voting Rate from 93.00% to 98.65%, outperforming LLM-decided structural evolution even with fixed budgets of 1, 3, or 5 rounds.
• Action-distribution analysis: split was the dominant structural action across both backbones, followed by save, with merge/revise/add occurring less often; GPT-5 mini triggered more revise/add actions than DeepSeek V3.2.
• Human/Gold Project reference scores (upper bound): ~100% Coverage and ~94–96% Pass Rate across the three repositories, indicating remaining headroom versus human-built repos.

Why it matters / caveats: The results support the paper's core claim that repository architecture should be treated as continuously evolving rather than a fixed one-shot plan, and that explicit modularity metrics (cohesion/coupling) outperform unconstrained LLM-driven restructuring, which tends to over-decompose and hurt downstream correctness. Evaluation is limited to six Python repositories from the RepoCraft benchmark with two specific LLM backbones, and thresholds (e.g., split/merge cutoffs) were empirically tuned on separate held-out repositories.

Problem: The quadratic cost of self-attention makes the prefill stage of long-context LLM serving a major bottleneck, and the authors' prior work FlashPrefill remains an algorithmic prototype—its accuracy degrades uncontrollably under aggressive sparsity, its FlashAttention-2-based kernel lags behind FlashAttention-3/4 on Hopper GPUs, and its contiguous KV layout is incompatible with paged KV cache and continuous batching used in production serving frameworks.

Method: FlashPrefill V2 extends FlashPrefill's block-level score estimation and max-based dynamic thresholding with three additions: (1) a zero-order mean correction term that compensates pruned key/value blocks with their pooled statistics (k̄_J, v̄_J) inside the online-softmax numerator/denominator, controlling approximation error under extreme sparsity; (2) a Hopper-aligned sparse attention operator built on CUTLASS/CuTe with PackGQA memory access, warp-specialized producer-consumer pipelines, intra-warpgroup pingpong GEMM-softmax overlap, index-driven (CSR) sparse traversal, single-pass fused scoring/selection, sparsity-aware load balancing, and FP8-e4m3 execution, matching the FlashAttention-3/4 execution model; and (3) native paged KV cache and variable-length/continuous-batching support, integrated as a standard attention backend into SGLang. Evaluation uses RULER and LongBench on Llama-3.1-8B-Instruct, Qwen3-4B-Instruct-2507, and Qwen3-30B-A3B-Instruct-2507, benchmarked against Full Attention, MInference, FlexPrefill, XAttention, and the original FlashPrefill on NVIDIA H20 GPUs.

Key results:

• Operator speedup over FlashAttention-2 at 128K context: 27.19× (BF16) and 47.26× (FP8); vs. an FA3/4-aligned dense baseline: 17.54× (BF16) and 30.49× (FP8).
• Outperforms prior sparse operators at 128K/batch=4: 18.67× (FlashPrefill v1), 5.43× (FlexPrefill), 3.42× (XAttention), 2.78× (MInference), vs. V2's 27.19×/47.26×.
• RULER average scores stay within ~1 point of Full Attention: 87.79 vs 88.82 (Llama-3.1-8B), 86.23 vs 87.06 (Qwen3-4B), 91.76 vs 92.05 (Qwen3-30B-A3B); at 128K (density <5%) gap widens to ≤1.8 points.
• LongBench: V2 has the highest average of all sparse methods on every model — 49.31 (Llama-3.1-8B) vs. 48.25 best baseline (XAttention); 46.96 (Qwen3-4B) vs 45.90; 50.73 (Qwen3-30B-A3B) vs 49.61.
• Attention density (Table 3) drops from ~70-76% at 4K to ~4.6-4.9% at 128K.
• Ablation of mean correction (Qwen3-4B, RULER): removing it costs up to 6.2 points at 128K in FP8; corrected pipeline stays within 1.5 points of full attention even at 128K in BF16.
• Threshold sweep at 64K FP8: density ranges 5.2%–23.6% as α goes from 0.2 to 0.0125; corrected pipeline stays within 2.7 points of full attention (82.81) across the whole range, vs. 2.7–5.4 points worse without correction.
• SGLang end-to-end TTFT reduction at 128K: up to 3.4× (BF16) and 4.8× (FP8); e.g., 123.2s → 36.2s (BF16) → 25.5s (FP8) at batch size 16 on Qwen3-30B-A3B.
• Open-loop serving (Qwen3-30B-A3B, 1 req/s): P50 TTFT speedup 5.14× (BF16), 11.44× (FP8); request throughput rises from 0.31 to 0.70–0.88 req/s.
• vs. production kernel HPC-Ops BSA (FP8, 64K): FlashPrefill V2 is 6–7% faster at every sparsity level, with comparable dense baselines (123.7ms vs 133.4ms), isolating the sparse-path gain.
• Mean correction runtime overhead at 64K: ≤12.2ms in the dense limit, only 3.6–4.5ms at 90% sparsity (5–14% relative overhead up to 50% sparsity, 18–27% at 90% sparsity).

Why it matters / caveats: The paper targets the production gap left by prior training-free sparse attention research—accuracy collapse at extreme sparsity and incompatibility with paged KV/continuous batching—making sparse prefill attention deployable in real serving stacks (demonstrated via SGLang integration) rather than remaining an isolated kernel benchmark. Caveats: gains are measured specifically on NVIDIA H20 GPUs; decode still falls back to dense attention (sparsity only applies to prefill); chunked prefill erodes speedup at short chunk sizes (recommends ≥8K chunks); and FP8 results marked with * use direct online quantization without corrected weights, showing a larger accuracy gap than BF16 in some cases.

Problem: Large language models often cannot answer questions about a bounded document collection at inference time when the source documents are not retrieved, and existing approaches (SFT on QA pairs, or continued pretraining) either provide sparse factual coverage or fail to translate document exposure into question-answering ability without damaging general capabilities.

Method: The paper proposes IAR (Inject, Align, Recover), a three-stage post-training framework: Inject exposes the model to source documents via continuation, rewrite (skeleton/outline reconstruction), and instruction-conditioned reconstruction objectives (mixed in configurable ratios, e.g., 1:1:1, 1:1:2, 1:0:0); Align fine-tunes the injected checkpoint with answer-only supervised QA loss; Recover merges the domain-adapted checkpoint back with the original base instruction model using post-hoc weight-space merging operators (SLERP, task arithmetic, TIES, DARE), selecting a checkpoint via a domain-primary frontier criterion on a validation split. It is evaluated on Common Corpus (CC, PleIAs 2024) and CCI (BAAI 2024) across Llama-3.2-3B, Phi-4-mini, Qwen3-4B, and SmolLM3-3B, with additional Qwen3-8B/14B/32B scaling ablations, against baselines including Vanilla SFT, BudgetMatch (token-budget-matched QA-only SFT), SDFT, LoRA, Replay, Base-initialized CPT+SFT, and FAPM.

Key results:

• IAR improves over Vanilla SFT on all four reported metrics (domain accuracy, IFEval, MMLU, MSBench) in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance.
• Qwen3-4B on CC: IAR reaches 50.5% domain accuracy vs. 42.4% for Vanilla SFT, while also improving IFEval (59.8 vs 51.1), MMLU (19.5 vs 8.8), and MSBench (63.0 vs 51.0).
• Qwen3-4B on CCI: base model already scores 70.6% domain accuracy; IAR still improves all four metrics (domain 76.3% vs Vanilla SFT 75.1%, IFEval 76.1 vs 45.6, MMLU 64.5 vs 26.3, MSBench 70.0 vs 49.5).
• Against BudgetMatch (token-budget-matched QA-only SFT), IAR wins on higher domain accuracy in 3 of 4 settings and higher mean general score in all 4 settings (14 of 16 metric comparisons).
• CC dataset: 14,258 training / 750 test QA pairs; CCI: 10,926 training / 575 test QA pairs.
• Qwen scaling (CC, RQ4): selected TIES (d=0.3) checkpoints stay within 1.1 points of Best pre-recovery (IA) domain accuracy while recovering 14.9–24.1 points in mean general performance (8B/14B/32B).
• LLM-judge evaluation reliability: across 242,255 evaluated instances, first-two-judge exact agreement of .707, binary agreement .848, Cohen's κ = .691, third-judge trigger rate .297.
• CC-only extended baselines: LoRA and FAPM can win individual general metrics, but IAR is domain-best for Phi, Qwen3-4B, and SmolLM3-3B, and second only to SDFT for Llama.

Why it matters / caveats: IAR offers a decomposition-based approach to trade off domain-specific accuracy against general capability retention when retrieval is unavailable (e.g., for latency, privacy, or testing parametric knowledge changes), but the authors note it is "not a recipe that dominates every baseline" — the best Inject mixture is model- and corpus-dependent, some settings (e.g., CC Llama, Phi CCI) show genuine trade-offs rather than uniform wins, and recovery remains partial rather than complete relative to the original instruction model's general scores.

Problem: Finetuning large vision-language-action (VLA) policies to learn new long-horizon, compositional, or reasoning-heavy manipulation tasks is difficult because collecting teleoperation data is expensive and reinforcement learning (RL) is too sample-inefficient, especially given the size and architecture of VLA models.

Method: Eximo is a three-stage pipeline — Explore, Imitate, Optimize — built on the 3B-parameter Gemini Robotics On-Device (GROD) VLA (a PaliGemma-backbone, diffusion-policy-head model trained on ALOHA teleoperation data). In Explore, a Gemini VLM acts as a closed-loop orchestrator that decomposes a long-horizon natural-language goal into short subgoals for the VLA, and successful rollouts (per a ground-truth success detector) are kept in a data buffer. In Imitate, the base GROD policy is supervised-finetuned (behavior cloning on action chunks, following Zhao et al. 2023) on this filtered orchestrated data, conditioned on the true task goal rather than the VLM's intermediate goals. In Optimize, a residual policy is trained with off-policy RL (MPO, Abdolmaleki et al. 2018) on top of the finetuned VLA's actions (a = a_VLA + Δa), following the residual RL approach of Ankile et al. (2025a).

Key results:

• Evaluated on 22 ALOHA-simulation manipulation tasks (dish-placement, reasoning, spatial/caddy, and chained-tool tasks), each tested over 1000 episodes.
• VLM orchestration increases GROD's success rate over the unorchestrated base VLA while maintaining similar time-to-success, with the largest gains on long-horizon (e.g., PlateBowlOnRack) and reasoning (e.g., BananaInBowl-Reasoning) tasks; orchestrated episodes are also shorter (more data-efficient).
• SFT on VLM-orchestrated data ("GROD + SFT") outperforms both the base GROD and GROD + VLM-Orchestration at evaluation time (no VLM needed at deployment).
• In online RL, GROD + SFT starts at a higher success rate and converges higher than the base GROD model, even when the base model is run for more environment steps (up to ~2.5e7) to compensate for Eximo's extra exploration data.
• SFT + RL (full Eximo) consistently outperforms both GROD + RL and GROD + SFT alone across all 23 task groupings; on several tasks, SFT alone (no RL) already beats the RL-finetuned base VLA despite the latter using far more data.
• Ablation: free-form VLM instructions perform on par with a grounded Pick&Place-only VLM baseline across 5 tasks, showing GROD needs no additional action grounding.
• Ablation: distilling VLM orchestration directly into the residual policy (via AWBC offline RL, then online RL) underperforms pure online residual RL, attributed to distribution shift between VLM-orchestrated and on-policy data.

Why it matters / caveats: The results suggest VLMs can substitute for teleoperators as a source of guided exploration data, letting VLA policies acquire new compositional/reasoning skills without added human teleoperation hours, and that distilling this data back into the VLA (rather than keeping the VLM in the loop at test time) improves both efficiency and performance. Caveats: evaluation is simulation-only (ALOHA), relies on a ground-truth success detector rather than a learned one, and residual-policy distillation of VLM guidance was found not to work well due to distribution shift — flagged as future work along with VLM-based success detection and environment resets.

Problem: Standard accuracy benchmarks fail to capture what actually changes when a frontier reasoning model is fine-tuned to think in a low-resource language (Greek), since accuracy is dominated by training noise and cannot see which language the model reasons in, what it spends, or what it forgot.

Method: The authors fine-tune three sparse mixture-of-experts model families at matched active-parameter budgets (3.6–4.0B active) — Qwen3.6-35B-A3B (Alibaba), Gpt-OSS-20B (OpenAI), and NemotronH-30B-A3B / Nemotron-3.5-Lightning-30B-A3B (NVIDIA) — using LoRA (r=32, α=64, one epoch, effective batch 32, lr 2×10⁻⁴) on a 118,092-row Greek corpus split into a reasoning half (59,107 rows, LLM-generated traces gated by the STaR answer-correctness rule) and a direct/instruction half (58,985 rows, reused from Sophea-Titan-1). They define six behavioural metrics (correctness, language fidelity, reasoning budget, termination, reasoning steps, budget overrun), each gated against correlation with trace length, and run a seed-variance control before ablations. A final stage applies GRPO-based RLVR (reinforcement learning with verifiable rewards, no KL term, 8 completions/prompt, lr 10⁻⁶) with five deterministic reward terms (correctness, language consistency, format, termination, override obedience) plus a random-reward control arm, pre-registered before training.

Key results:

• Seed-only variance on a 35B MoE LoRA run moved accuracy by 7.7 points, larger than every data/recipe intervention tested (sd=4.4pp over seeds 42–44).
• Base models never reason in Greek (0/1,000 traces reach Greek ratio ≥0.9); after SFT, all four released checkpoints reason in Greek on 97.4–98.1% of items (up to 98.7% for a one-directional variant), with no accuracy loss (best arm 76.5 vs base 77.2 on the 1,000-item probe).
• SFT cuts reasoning cost 3.7–5× in words but token cost changes sign by family: Qwen spends 3× fewer tokens (586 vs 1,788), NemotronH ~parity (638 vs 681), Gpt-OSS 1.6× more (640 vs 396) due to Greek's 2.3–2.5× token fertility.
• Apparent forgetting mostly wasn't: a −3.0pp commonsense "regression" reversed to +1.7pp under constrained-format re-scoring; register control and grammaticality improved on all four families; general ability (Titan-1 suite) stayed flat on two families, with only a −3.2pp Greek residual on Gpt-OSS after format repair.
• SFT could not fix its own defects: answer-format fallback 24% and answer-channel leak 3.5% (Qwen); explicit "think in English" instructions obeyed only 44.8% (Qwen) / 62.5% (Gpt-OSS) / 0% (both Nemotron models).
• Pre-registered RLVR (GRPO) fixed these: fallback 24.1%→2.5%, leak 3.53%→0.00%, both against a flat random-reward control (22.1%, 3.61%); override obedience rose 44.8%→53.9% (+9.1pp, +9.8pp control-adjusted), short of the pre-registered ≥15pp/60% trainability bar; Greek-trace fidelity survived an accuracy-only gradient at 98.2–98.27%.
• Six documented instrument failures include: a +29.8pp (18σ) scorer artifact from an un-anchored answer extractor; 38.9% contamination in the logic probe (175/450 items) undetected by standard checks; a locale bug misreading Greek-format numbers (17.500, 3,5).
• One accuracy effect survived the noise floor: dropping the non-reasoning corpus half (reasoning-only recipe vs two-phase) gave +6.9pp over 15 independently trained arms (permutation p=0.0008).

Why it matters / caveats: The paper argues that for low-resource-language reasoning fine-tunes, single-run accuracy deltas are largely noise and behavioral instrumentation (language fidelity, cost, forgetting, steerability) is what actually reveals fine-tuning effects; results are explicitly scoped to sparse MoE models adapted via LoRA and may not generalize to dense architectures or full fine-tuning. Five checkpoints are released (Sophea-Qwen3.6-v1/v1.1, Sophea-OSS-v1, Sophea-Nemo-3-Nano-v1, Sophea-Nemo-3.5-Lightning-v1).

Problem: Public ASR benchmarks can be gamed—models may drive down reported word error rate (WER) by exploiting benchmark-specific artifacts (transcription errors, masked or ambiguous audio, orthographic conventions) rather than by genuinely improving general-purpose transcription ability, and no existing methodology quantifies this "benchmark optimization" (benchmaxxing) phenomenon.

Method: The authors introduce three behavioral probes—reference disagreement, masked-number recovery, and orthographic switching—that measure how often a model reproduces a benchmark's reference transcript ("accept-ref") even when the audio contradicts, omits, or is ambiguous about that content, plus a white-box "audio lift" metric (Eq. 1) comparing teacher-forced likelihood of the reference span under real audio versus silenced audio. They evaluate 11 open-source ASR models spanning encoder-decoder/transducer architectures (Whisper-Large-v3, Cohere-Transcribe, Parakeet-TDT-0.6B-v2, Moonshine-Streaming) and speech-LLM architectures (Canary-Qwen-2.5B, Granite-Speech-4.1-2B, Higgs-Audio-v3-8B, Kimi-Audio-7B, Phi-4-Multimodal, Qwen3-ASR-0.6B, Voxtral-Mini-3B) on VoxPopuli and LibriSpeech, with held-out controls DaiKon (450 private conversational clips), ep-fresh (freshly scraped June 2026 European Parliament recordings), and libri-fresh (2026 LibriVox recordings from new readers), plus Qwen3-TTS-generated voice clones. Reference errors are mined via a consensus panel (Kimi-Audio, Qwen3-ASR-0.6B, Voxtral-Mini-3B, Moonshine-Streaming) validated against human annotations. Mechanistic localization uses activation patching, an attention-mask/translation readout, and diff-in-means linear steering trained on 22 ep-fresh courtesy-opener splice pairs applied at a single encoder layer.

Key results:

• The six models with best VoxPopuli WER (5.4–5.8%) have the highest reference-disagreement accept-ref (0.18–0.30); every model at ≥6.5% WER scores ≤0.10.
• Consensus panel flagged 1,113 reference edits across 745 VoxPopuli test clips (586 substitutions, 441 deletions, 86 insertions); 93% match human annotations; ~3% of all reference words carry a flagged edit, affecting 40% of clips.
• Masked-number accept-ref on public benchmarks reaches ~0.40 for top models, higher than on held-out libri-fresh/ep-fresh.
• 6/11 models exceed the 0.5 baseline switch rate on the honorific convention; 8/11 exceed it on archaic spacing.
• No leaked-speaker advantage found: pooled elevated-model reference-disagreement accept-ref is 0.22 on leaked vs. 0.26 on unleaked VoxPopuli speakers.
• Appending a VoxPopuli "donor" clip to low-accept-ref ep-fresh clones raises accept-ref (Phi-4 +.10, Canary +.09, Higgs +.07, Cohere +.07, Parakeet +.04); a conversational donor collapses accept-ref on real benchmark clips for Cohere, Canary-Qwen, Phi-4, Parakeet, and Higgs.
• Linear steering (k=1 direction) recovers 65–80% of full-direction effect for Cohere, Parakeet, Canary; adding it to generic audio raises accept-ref (e.g., Cohere 0.02→0.07, Canary 0.01→0.11); ablating it drops accept-ref by 82–92% for Cohere, Canary, and Parakeet.
• On hard cells (silenced-audio NLL/char ≥3.5 nats), masked recovery rises for elevated models (Cohere 0.27→0.36, Canary 0.14→0.24, Higgs 0.15→0.23, Granite 0.10→0.19) but stays near floor for Moonshine (0.00), Kimi/Qwen3 (0.07).
• Models with high accept-ref (Phi-4, Cohere-Transcribe, Granite, Canary) reportedly trained on less than 1 million hours of data, while low-accept-ref Qwen3 trained on 40 million hours of weakly supervised data.

Why it matters / caveats: The findings suggest high WER on public leaderboards can partly reflect narrow, benchmark-specific acoustic shortcuts rather than genuine transcription improvement, particularly since ASR models' bidirectional audio attention gives them more surface area for such shortcuts than text-only LLMs; the authors recommend held-out/non-public test sets, temporal/speaker stratification, and multi-metric evaluation beyond WER. A caveat is that the mechanistic steering results only causally generalize to a subset of models (4 of 6 "elevated" models, with Phi-4's direction causally inert and Higgs-Audio showing no clean steering operating point).

Problem: LLM customer-service agents often violate organizational policy not just by taking forbidden actions but by skipping or misordering required procedural steps (identification, eligibility checks, confirmation), and existing action-local safeguards only catch this at the final mutating call, too late to guide the agent through the earlier missing steps.

Method: PolicyGuide compiles each domain's natural-language policy into a frozen workflow graph (offline, via a six-stage GPT 5.4-driven pipeline: extract tools/mutating actions, derive request types and subflows, review, generate and schema-validate subflows, connect intake/classifier/subflows, then validate schema conformance, tool coverage, and reachability). At runtime, an external proactive verifier LLM (model-paired with the agent) fires at user-turn boundaries and after intercepted unauthorized mutating calls, reconciles all open requests against code-persisted graph state, traverses each request from its saved node to the first unsatisfied requirement, and returns targeted remediation plus updated state and mutation-gate status — decoupling workflow tracking (external, code-owned) from the acting agent. It is evaluated against ReAct (no guard), ToolGuard, PolicyGuard, and FlowAgent baselines on τ²-bench Airline, Retail, and Telecom domains, with GPT 5.4, Claude Sonnet 4.6, and Gemini 2.5 Pro as agents and a frozen GPT 4.1 user simulator.

Key results:

• Mean Pass4 across the three τ²-bench domains rises from 0.42 (unguided ReAct) to 0.62 with PolicyGuide (GPT 5.4 agent/verifier), largest gain on Telecom: 0.19 → 0.61 (later reported as 0.614 in Table 1).
• Domain Pass4 (Overall/PV/Mut): Airline 0.620/0.917/0.346, Retail 0.614/0.900/0.587, Telecom 0.614/0.721/0.549 — highest overall Pass4 in all three domains versus ReAct, ToolGuard, and PolicyGuard.
• Matched workflow-controller comparison on Telecom (40-task test split, Pass4): ReAct 0.250, PolicyGuard 0.325, FlowAgent (PDL + API control) 0.350, PolicyGuide (external graph verifier) 0.675.
• Cross-agent transfer of the same GPT 5.4-authored Airline workflow: Claude Sonnet 4.6 Overall Pass4 0.720→0.780; Gemini 2.5 Pro 0.480→0.680, with Mut Pass4 improving from 0.231 to 0.462.
• CRAFT adversarial red-teaming (Airline, 20 attack tasks): PolicyGuide has the lowest attack-success rate at every k, per-trial ASR 0.087 vs. 0.125 (PolicyGuard) vs. 0.200 (ReAct) — preventing 91.3% of tested attacks.
• Author-designed Telecom ordered-trace audit: process-valid rate 56.2% for PolicyGuide vs. 17.5% (ReAct) and 13.1% (PolicyGuard); Trace-TCR 63.4% vs. 35.4% and 23.9%.
• Source-policy analysis: procedural requirements are pervasive (67.4% airline, ~100% retail, 98.0% telecom); ordered workflow requirements concentrate in telecom (54.0% vs. 4.7% airline, 3.6% retail).
• Ablations (Telecom test split, Pass4): full PolicyGuide 0.675 vs. PolicyGuide-Self (graph given to actor, no external verifier) 0.325 and PolicyGuide-Raw (raw policy text instead of graph) 0.350, showing both the compiled graph and external tracking matter.
• Estimated cost: ~$0.40 per conversation for guide calls.

Why it matters / caveats: Treating the whole procedure — not just the final action — as the unit of policy compliance yields large, transferable gains, especially in workflow-heavy domains like Telecom, without requiring model retraining. The authors caveat that node judgments are probabilistic (not a formal guarantee), verifier exceptions are fail-open, evaluation is limited to three English τ²-bench domains with a frozen simulated user, and the Telecom trace-compliance rubric is an exploratory, author-designed metric without inter-annotator agreement.

Problem: Conventional LLM evaluation and deployment treat every inference as an isolated, zero-shot event, ignoring the model's potential to accumulate and learn from experiential feedback across iterative interactions at test time.

Method: The authors introduce Chain-of-Experience (CoE), a framework in which a model's response at each step is conditioned on the full history of prior actions and feedback (a sequential generative process at ~ P(at | Q, e0,...,et-1), where each experience ei = (ai, fi)). CoE is instantiated with four feedback types spanning a richness spectrum: no feedback, execution feedback (from code interpreters/unit tests), model feedback (an auxiliary LLM acting as judge/critic), and correctness feedback (binary oracle signal). The framework is evaluated training-free (no parameter updates) across math (AIME 2025, OmniMath), coding (LiveCodeBench V6, LiveBench-Code), and knowledge (EvaLearn, GPQA Diamond) benchmarks using 8 reasoning LLMs (GPT-5, GPT-5-mini, o4-mini, o3, o3-mini, Gemini-2.5 Pro, Claude 4.5 Sonnet), run for up to 20 iterations, and compared against baselines including built-in reasoning-effort scaling, few-shot ICL, Dynamic CheatSheet (DC), and Agentic Context Engineering (ACE).

Key results:

• Overall: self-feedback CoE gives +5.6% average improvement with 19% lower API cost versus no-feedback baseline; correctness/executor feedback gives +11.1% (upper bound).
• Averaged across six benchmarks (Table 3): ICL 62.1%/62.9%, ACE 64.0%, DC 62.7%, no-feedback 66.8%, self feedback 71.0% (+7-9% over ICL/ACE/DC), best feedback (correctness/executor) 79.3%.
• Coding tasks: executor feedback lifts accuracy from 66.4% to 75.0% (+8.6%); self feedback gives +7.0%.
• Non-coding tasks: 75.1% (correctness) > 67.1% (self) > 62.5% (no feedback).
• Efficiency: self feedback captures most of executor's gain on coding tasks (73.4% vs. 75.0%) with 13.4% fewer API calls ($70.7 vs $81.6) and 20% less cost than no-feedback; on AIME 2025 and EvaLearn, self feedback cuts cost by 47.3% and 7.0% while improving accuracy by 4.4% and 6.9%.
• Token efficiency: on AIME 2025, correctness feedback reaches 84.6% using 108K tokens vs. DC's 74.7% at 11K tokens and no-feedback CoE's 74.1% at 107K tokens.
• Improving capability correlates with base ability: average Pearson r = 0.50 across benchmarks (LiveBench-Code r=0.97, LiveCodeBench r=0.83, AIME 2025 r=0.33, OmniMath r=0.24).
• Spurious feedback: constant incorrect/correct feedback degrades performance by 7.6% (AIME 2025) and 2.6% (GPQA Diamond) on average, but GPT-5-mini drops only 2.5%/0.6% vs. o4-mini's 12.8%/4.6%. Selective Majority Voting (SelMV) recovers robustness (e.g., 79.4%→80.3% on GPQA Diamond).
• Improvement-pattern analysis (6,630 incorrect→correct pairs, judged by GPT-5, 76.8% agreement with human ratings): 47.7% of improvements are feedback-driven; 30.0% of coding improvements stem from specification recall; model-generated feedback yields higher feedback-attributed improvement than other sources (58.7% vs. 41.1%).
• Dual feedback (model + correctness/executor, Claude 4.5 Sonnet): AIME 2025 76.7% vs. 70.0% (correctness-only)/60.0% (model-only); LiveBench-Code 81.2% vs. 78.1%/57.8%.
• Memory-compression baselines underperform full experience trails: on AIME 2025, model feedback alone 60.0% vs. 50.0% (+DC) and 56.7% (+SimpleMem); on LiveBench-Code, 57.8% vs. 51.6% (+DC) and 54.7% (+SimpleMem).

Why it matters / caveats: CoE demonstrates that simple, training-free feedback loops can outperform more elaborate cross-task memory methods (DC, ACE) and reduce inference cost simultaneously, suggesting test-time feedback accumulation is an efficient lever for LLM improvement without retraining. Caveats: correctness/oracle feedback is often unrealistic in real-world deployment; benefits scale with base model capability (weaker models gain less and are less robust to spurious feedback); and aggressive memory compression can discard useful intermediate reasoning, making the value of memory-based selection methods within-task unclear.

Problem: Existing long-video benchmarks rarely jointly evaluate sustained narrative comprehension and culturally implicit reasoning in high-context, non-English (specifically Japanese) long-form media.

Method: The authors build NARU using a hierarchical memory-based annotation pipeline: videos are chunked into ~5-minute units (Step 1-1), then re-segmented into chapter-level segments via lower-rate visual sampling guided by Segmented Discourse Representation Theory (Step 1-2), then annotated with task-oriented narrative and cultural evidence across a taxonomy of four narrative categories (N.1–N.4, grounded in Event Segmentation Theory) and five cultural categories (C.1–C.5, e.g., Aizuchi, Kuuki wo Yomu, Subtext Interpretation, Cultural Context Recognition, Sentiment Analysis) (Step 1-3). Multiple-choice QA pairs are generated via capability-conditioned formulation and refined through an iterative "Solver-Critic Loop" (Blind Solver Agent, Diagnostic Agent, Question Revision Agent) to remove text-only shortcuts; Gemini 2.5 Pro is used for all MLLM-based pipeline components. Final items go through two human verification stages by 68 native Japanese annotators. Evaluation covers eight MLLM configurations (Gemini-3-Flash, Gemini-3-Pro, Gemini-2.5-Flash, Qwen3.5-9B, Qwen3-VL-8B, Qwen2.5-VL-7B, MiniCPM-o-2.6, InternVL3.5) under both multiple-choice and open-ended (judged by GPT-5.5 using a FActScore-style atomic-fact recall metric) settings.

Key results:

• NARU: 1,481 QA items grounded in 155 Japanese videos totaling 146.8 hours (745 narrative + 736 cultural questions), sourced from an initial pool of 100,000+ YouTube videos filtered down to 8,018 candidates.
• Verification stats: of 1,500 candidate questions, 178 flagged invalid (161 repaired, 17 removed) → 1,483 progressed; post-refinement, 949 accepted unchanged, 532 revised, 2 removed → final 1,481 items.
• MCQ accuracy: Gemini-3-Flash highest overall at 76.2%, Gemini-3-Pro 70.0%, Gemini-2.5-Flash 51.4%; open-source models ranged 29.6–39.8% (Qwen3.5-9B best open-source at 39.8%).
• Gemini models scored ~11 percentage points higher on narrative than cultural tasks; open-source models showed little narrative/cultural gap.
• Subtext Interpretation (C.3) was hardest even for frontier models — Gemini-3-Flash dropped to 57.4% on C.3 despite a 68.2% cultural average.
• Several open-source models fell below the 25% random-guessing baseline on N.1, C.1, and C.4.
• Frame-budget sweep (8→128 frames, 500-question subset): Gemini-3-Pro rose from ~64% to ~71%, Gemini-3-Flash 53%→64%, Gemini-2.5-Flash 39%→51%; open-source gains were smaller/inconsistent (e.g., Qwen3VL-8B ~31%→38%, others near 30% even at 128 frames).
• Narrative accuracy gains from more frames ranged +1.4 to +20.5 pp; cultural accuracy changed from -3.9 to +10.0 pp, showing narrative understanding scales with visual context while cultural understanding does not.
• Open-ended (FActScore recall) results: Gemini family 0.66–0.78 recall vs. 0.21–0.56 for open-source models; N.2 (Sequential/Topical Flow) flips from the easiest MCQ category to the weakest open-ended category for 7 of 8 models; narrative vs. cultural difficulty ranking also reverses between MCQ and open-ended formats (78.9% of cultural responses recovered ≥1 reference fact vs. 66.3% of narrative responses).
• Human agreement with the automated GPT-5.5 judge: 90.0% acceptance across 150 verdicts (individual annotator rates 88–92%), majority-vote acceptance 96.0%, mean pairwise annotator agreement 82.7%.

Why it matters / caveats: The results show open-source MLLMs lag substantially behind proprietary Gemini models on both long-range narrative tracking and culturally grounded reasoning in Japanese long-form video, and reveal that narrative and cultural understanding rely on distinct mechanisms (visual context density vs. pre-training knowledge), motivating NARU as a diagnostic benchmark for developing MLLMs with genuine high-context video understanding.

Problem: Existing self-supervised audio representation learning methods have grown increasingly complex, relying on reconstruction decoders, acoustic tokenizers, student-teacher EMA setups, and auxiliary regularization losses, while the autoregressive next-embedding prediction paradigm that has driven progress in language modeling and vision remains unexplored for audio despite audio's inherently sequential temporal structure being a natural fit for it.

Method: The paper introduces NAPE (Next-Audio-Patch-Embedding prediction), which splits a log-mel spectrogram into non-overlapping 16×16 patches via a Conv2d patch embedding layer, linearizes the resulting 2D patch grid into a 1D sequence (comparing raster, time-major, zigzag, and diagonal scanning orders), and trains a causal Vision Transformer encoder (with RoPE, LayerScale, and query-key normalization) plus a lightweight SimSiam-style 3-layer MLP predictor head to predict each next patch embedding from preceding patches. The sole training signal is negative cosine similarity between predicted and stop-gradient target embeddings (following SimSiam), with no reconstruction decoder, tokenizer, EMA teacher, or auxiliary losses; NAPE is pre-trained on AudioSet (unlabeled, ~1.96M unbalanced + 20,961 balanced clips) at three scales — Small (~19M params), Base (~85M), Large (~303M) — using AdamW with cosine LR decay for 25–30 epochs, and fine-tuned/linear-probed on AS-2M, AS-20K, ESC-50, Speech Commands V1/V2, and IEMOCAP.

Key results:

• NAPE-L (raster) reaches 50.2 mAP on AS-2M (tying top baseline SSLAM), 40.5 mAP on AS-20K (vs SSLAM's 40.9), and 96.0% on ESC-50 (vs SSLAM's 96.2%).
• On IEMOCAP, NAPE-L reaches 68.0–68.8% accuracy, a +3.5-point improvement over the strongest reported baseline (BEATs iter3 at 64.5%).
• NAPE outperforms Audio-MAE at every scale, with the largest margins at Small size: +2.6 mAP on AS-2M, +4.1 mAP on AS-20K.
• Ablations: removing prediction shift or stop-gradient causes training to diverge; removing the causal mask (keeping shift) degrades AS-2M by −7.8 mAP, AS-20K by −14.3 mAP, ESC-50 by −25.4 points; Conv2d patch embedding beats Convstem (−4.8 mAP on AS-20K) and Speechstem (−6.0 mAP on AS-20K); SimSiam-style predictor beats a Transformer predictor despite being ~8× smaller (1.8M vs 14.2M params); raster/diagonal/zigzag scanning orders perform comparably while time-major consistently underperforms; using raw-mel or 1st-encoder-layer targets either matches or destabilizes (diverges) training relative to the default patch-embedding target; L1/L2 similarity losses diverge, cosine similarity performs best.
• Linear probing improves with scale (AS-2M: 23.2/25.0/27.1 mAP for Small/Base/Large; ESC-50: 79.8/81.7/83.5%), with the best probing layer at roughly the middle of the encoder (layer 2/6/11 for Small/Base/Large).

Why it matters / caveats: NAPE shows that a minimalist causal next-embedding objective — without decoders, tokenizers, teacher-student distillation, or regularizers — can match or approach much more elaborate state-of-the-art audio SSL recipes (e.g., SSLAM, A-JEPA) while scaling favorably across model sizes, suggesting audio's temporal structure makes it particularly well-suited to the autoregressive predictive paradigm already dominant in language and emerging in vision. A caveat is that linear-probing accuracy remains notably lower than fine-tuning results, since the predictive objective is not inherently aligned with linear separability.