AI papers — 2026-08-11

Problem: Existing code-agent benchmarks like SWE-bench Verified are saturating and suffer from evaluation-quality defects (nearly 60% of unsolved instances have flawed tests) and Python-centric, small-scope tasks, leaving large-scale multilingual code refactoring underserved as a harder, more realistic test of agent capability.

Method: The authors build SWE-Bench ProMax, 170 expert-curated refactoring instances mined from real commits across seven languages (Python, Java, TypeScript, Go, C, C++, Rust) via a three-stage pipeline: (1) GitHub API collection of commits from repos with ≥500 stars, approved licenses, and ≥80% target-language code, filtered for the keyword "refactor" (excluding "bug fix") and modifications to both test and non-test files, from an initial pool of 29,782 candidates; (2) Docker environment construction (via SWE-Factory) with the gold patch applied and full test suite executed to validate instances; (3) human-expert curation with LLM assistance to rewrite issue descriptions from scratch as precise, self-contained specifications, remove overly narrow/broad tests, and verify necessity/sufficiency of each description relative to the gold patch. They evaluate six frontier models (Gemini-3-Pro, Claude Sonnet 4.6, GPT-5.2, GLM-5, Kimi-K2.5, Qwen3.5) under two agent scaffolds (mini-swe-agent and OpenHands), with a 300-step and $10-per-instance cap, measuring Pass@1 resolve rate.

Key results:

• Only 170 instances survived curation out of 29,782 initial candidates, spanning 70 repositories; gold patches average 11.4 modified source files and 261.6 lines of code (up to 182 files and 4,503 LOC), versus SWE-bench Verified where 86% of instances touch just one file.
• Best model overall is GPT-5.2 under OpenHands with 41.2% resolve rate — far below the 75%+ frontier agents achieve on SWE-bench Verified.
• Cost efficiency varies widely: Claude Sonnet 4.6 averages $4.77/instance for 38.8% resolve rate, while GLM-5 reaches a comparable 36.5% at only $0.24/instance (about 1/20th the cost); Qwen3.5 also hits 36.5%.
• Scaffold matters: switching from mini-swe-agent to OpenHands boosts GPT-5.2 from 21.8% to 41.2%.
• No single model dominates across languages (e.g., Claude Sonnet 4.6 leads TypeScript at 53.6% and Rust at 63.6%; GPT-5.2 leads C at 75.0% and Python at 48.3%; GLM-5 leads Java at 34.6%).
• Failure analysis: agents' modified-file-count CDF reaches 90% at ~10 files versus ~20 files for the gold patch, indicating agents under-modify relative to what's required; failed trajectories consume more interaction rounds than successful ones.

Why it matters / caveats: The benchmark shows multi-file, cross-language refactoring remains far from saturated (41.2% ceiling) even for frontier models, and demonstrates open-weight models can approach proprietary performance at a fraction of the cost, suggesting cost is not tightly coupled to capability on this task. The dominant failure mode identified — incomplete propagation of changes across files rather than poor localization — points to sustained cross-file coordination as the key bottleneck for current agents.

Problem: Post-trained LLM agents are typically frozen at release into a checkpoint tied to the task/environment distribution seen during training, with no systematic mechanism to turn deployment experience into ongoing improvement or to compose specialized skills without cross-task interference.

Method: Macaron-V1 combines a Mixture-of-LoRA (MoL) architecture — a frozen base model with four specialist LoRA adapters (Chat/L0, Agent/L1, Coding/L2, GenUI/L3) selected per user turn by the L0 adapter itself under a constrained-decoding router, no separate router model — with a recursive self-improvement (RSI) loop called MindForge that cycles through Discovery (harder task generation), Expansion (trajectory collection and harness/config search via a versioned Harness Context Protocol), and Update (GRPO training of the LoRA adapters, base frozen). The flagship Macaron-V1-Venti pairs a 744B GLM-5.2 base with four 1B-labeled LoRAs (rank 16, alpha 32, ~7.69B stored values per adapter); Macaron-V1-Tall uses the same design on a Qwen3.6-35B-A3B base (rank 64, alpha 128, ~3.78B values per adapter, ~50.1B total). Supporting infrastructure includes the MinT post-training platform (adapter-revision/policy-record separation), LongStraw for response-only long-context RL execution, and sparse-MoE/DSA rollout-training mismatch controls (R3 routing replay, DSA alignment, IcePop-style filtering). Evaluation uses two new Personal Intelligence benchmarks (Macaron ChatBench, 46 cases; Macaron LivingBench, 40 scenarios), UI4A-Bench (161 GenUI cases), and a general-capability suite (VitaBench, PinchBench, ClawGym, SWE-Verified, TerminalBench 2.1, etc.) against GLM-5.2, GPT-5.5, Opus 4.8, Gemini 3.1 Pro, Qwen 3.7 Max, and Minimax M3.

Key results:

• Routing: L0 reaches 99.12% accuracy (6391/6448) with 100% canonical-label compliance and zero parse errors; Macaron-V1-Tall reaches 99.04% on the same trace.
• Routing overhead is ~32% of a three-hop turn on Venti (route 0.54s + summary 0.97s vs. answer 3.17s) and ~30% on Tall (0.20s + 0.32s vs. 1.24s).
• MoL residency: storing one shared 744B base plus ~30.8B adapter values (~774.8B logical) is ~26.0% of the size of four replicated merged-base copies (2.976T), a 74.0% reduction.
• Fixed-model harness-search (RSI Expansion stage only, no weight update) reaches 122/122 cumulative coverage on a 122-task base-failure set (frozen GLM-5.2 fails all of them), whereas the two single full-set configuration sweeps reach only 4/122 (3.3%) and 11/122 (9.0%) — a 13x per-attempt yield gap between the final search phase (81.2% pooled pass rate) and those full sweeps.
• Benchmark results (0–100 scale): Macaron-V1-Venti scores 58.3 on ChatBench, 64.0 on LivingBench, 60.0 on VitaBench, 94.0 on PinchBench, 87.6 on TerminalBench 2.1, 85.6 on SWE-Verified, and 87.8 on UI4A-Bench (Final Score), generally at or above the six frontier baselines on most rows (e.g., GLM-5.2 base scores 67.1 on UI4A-Bench vs. Venti's 87.8).
• Macaron-V1-Tall outperforms its own Qwen3.6-35B-A3B base on all 7 matched benchmarks, e.g., UI4A-Bench 59.3 vs. 33.9, ChatBench 54.9 vs. 48.0, ClawGym 64.0 vs. 58.6.
• UI4A generative-UI harness reduces output tokens ~45% vs. raw HTML (~672 vs. ~1,224 tokens on a 48-case gallery) and gives up to ~6x faster time-to-first-render.

Why it matters / caveats: The paper is explicit that many headline numbers are architecture/infrastructure diagnostics rather than end-to-end causal evidence — e.g., the 122/122 coverage result only tests frozen-model configuration search (no adapter training occurred), the internal Personal Intelligence benchmarks are not independent of the training data pipeline, and claims about cross-team/cross-user "collective intelligence" composition remain untested architectural affordances rather than demonstrated results.

Problem: In-context learning and latent (non-verbalized) reasoning have developed separately — chain-of-thought models learn flexibly from context but spend tokens verbalizing computation, while compact recursive solvers like HRM/TRM reason iteratively in latent space but require task-specific optimization on evaluation-task demonstrations rather than acquiring an unseen transformation purely from in-context examples.

Method: BDH-CQ extends the Dragon Hatchling (BDH) architecture — a post-Transformer design using high-dimensional positive activations, low-rank ReLU transformations, and linear attention over a recurrent associative state — into a system where demonstrations sequentially update a recurrent memory state S_t = U_θ(S_{t-1}, D_t), and the query is then solved by R steps of iterative latent computation (H_{r+1} = F_θ(H_r, S_K)) in a structured workspace before a single decode step, with no parameter updates at inference time and no task identifiers. A 150M-parameter model was trained on a curated ARC-style mixture combining private data with ARC-AGI-1 training set, RE-ARC, ConceptARC, ARC-Heavy, and ARC-GEN100K, plus additional augmentations; it is evaluated on the 400-task public ARC-AGI-1 evaluation split under the leaderboard's pass@2 convention, and separately on ConceptARC and controlled/generated ARC-like task families (varying propagation distance, copy count, sequence length, nesting depth, operation composition, etc.) to probe generalization.

Key results:

• 29.5% pass@2 (118/400 tasks) on public ARC-AGI-1 at a computed cost of $0.00070/task (~0.85 H200 GPU-seconds/task at $3/hr), reported as breaking the previously known ARC-AGI-1 cost–accuracy Pareto frontier and independently reproduced by external auditors from Bielik and NYU.
• ~57x cheaper than GPT-5.6 Luna (Low), which scores 34.2% at $0.040/task per ARC Prize's July 2026 data (~11x cheaper after accounting for OpenAI's later 80% price cut); vastly cheaper than HRM ($1.48/task) and TRM ($1.76/task), which also require test-time optimization on eval tasks.
• On ConceptARC: 59.38% strict task pass@2 / 77.92% test-pair pass@2 (semantic IDs), nearly unchanged with opaque IDs and shuffled batches (60.00%/77.92%), ruling out a simple identifier/batch-context confound.
• Controlled generalization: boundary propagation and motif copying solve 48/48 held-out cases across tested ranges; ordering collapses from near-saturation to 1/24 at length 8 and nesting drops to 29/36 at depth 5, but adding one matched-complexity demonstration recovers nesting to 24/24 and ordering to 13/24 (from 0/24), showing failures trace to demonstration coverage, not raw capacity.
• Composition results (72 held-out outputs/condition): rotation+relocation solved 72/72, reflection+relocation 47/72, color-swap+relocation 0/72 (swap alone only 26/72, and only in one motif layout).
• Reasoning-effort scaling: pass@2 of 21% (LOW), 27% (MEDIUM), 29.5% (HIGH), with cost reductions of 22% and 11% respectively relative to HIGH.

Why it matters / caveats: The result demonstrates that recurrent latent reasoning can absorb in-context demonstrations without any inference-time weight updates or verbalized chain-of-thought, at extremely low cost, suggesting a scalable alternative to token-heavy CoT and to transductive solvers that require per-task optimization. Caveats: model architecture/dimensions and the full training recipe are kept proprietary; ARC-AGI-1 accuracy (29.5%) remains well below frontier LLM reasoning systems; behavioral analysis shows inconsistent within-task rule application (52/160 ConceptARC tasks partially but not fully solved) and clear failure modes on ordering, nested containment, and rule-selection/parameterization tasks.

Problem: Most agent harnesses are frozen after design, so long-horizon agent capability is bottlenecked by static context assembly, tools, and prompts rather than by the underlying model.

Method: Ouroboros is a coding-agent harness whose own source (tools, context assembly, prompts, core implementation) evolves through reviewed Git commits that become the runtime for later tasks, via two modes: recursive free evolution (improvement itself is a scheduled task) and experience-driven core evolution (bugs/rough edges surfaced by ordinary work or social feedback trigger structural fixes). Commits pass a deterministic preflight, staged-diff fingerprinting, and a multi-model adversarial diff-review panel with quorum (plus whole-repository scope review in "max" context mode), and self-repository mutation is bounded by owner-selected runtime modes (light/advanced/pro). The system is evaluated with frozen seeds on Terminal-Bench 2.1, OSWorld-Verified, CL-Bench, SWE-bench Pro, and GAIA using official verifiers and public run manifests, while a separate live lineage called "Hope" runs continuous free evolution under human interaction across seven communication surfaces (web, voice, Telegram, Discord, Twitter/X, website comments, email) since February 2026.

Key results:

• Terminal-Bench 2.1 (Opus 5, 89 tasks × 5 trials): 86.97% raw (387/445), 86.74% after trajectory audit (386/445) — best reported score, vs. Claude Code+Fable 5 at 83.8%, Codex CLI+GPT-5.5 at 83.1%, Cursor+Grok 4.5 at 79.3%.
• OSWorld-Verified (Opus 5, 361 tasks): 90.69% (327.39/361), above Intelligence-Indeed's leaderboard-leading 90.19% and Claude Mythos Preview's 85.4%.
• CL-Bench (Sonnet 4.6, 5 ordered rollouts): normalized reward 0.2301, new SOTA vs. ICL/Sonnet 4.6 (0.1960), ICL/GPT-5.4 (0.1890), Claude Code/Sonnet 4.6 (0.1855).
• SWE-bench Pro (GPT-5.6 Luna, 655 paired/contamination-filtered tasks): 58.2% vs. Codex 59.4% (McNemar p=0.40, statistically indistinguishable — model-matched parity).
• GAIA (Sonnet 5): 78.2% vs. Claude Code's 78.8%.
• Hope deployment (161 days as of Aug 6, 2026): $110.6K model spend, 79.7B tokens, 175,755 LOC, 227MB memory artifacts, ~3,600 participants, 222,474 public messages, 1,085 self-modification commits (94.2% agent-authored), 1,522 reviewed self-edit attempts with a 63.5% recent review block rate, and 40 pattern classes covering 659 recurrences.

Why it matters / caveats: Demonstrates that letting an agent reviewably rewrite its own harness (not just weights/prompts) can push state-of-the-art on several agentic benchmarks, and that a governed, long-running "living agent" can incorporate social feedback into structural self-improvement without ceding commit authority to users. Caveats: the deployment study is a single lineage rather than a controlled population; SWE-bench Pro still suffers reference leakage/task defects requiring symmetric filtering; LLM reviewers can share blind spots with the agent being reviewed; and low context mode skips whole-repository scope review, weakening self-modification oversight.

Problem: Small LLM agents struggle to build useful memory on their own because their low task success rates leave memory repositories dominated by failed trajectories, and naively transferring a large teacher agent's memory yields only marginal gains due to the teacher-student capability gap.

Method: The authors propose Agent Memory Distillation (AMD), a training-free framework where a teacher agent (GPT-5-mini) generates three hierarchical memory types from its successful trajectories: Workflow memory (natural-language, task-level strategy with runtime values replaced by typed placeholders), Subtask memory (code-centric execution examples for semantically coherent trajectory segments), and Function memory (per-function call examples plus API docs, indexed by function name). Workflow and Subtask memories are retrieved via dense embedding similarity (OpenAI text-embedding-3-small) and proactively injected into the system prompt before task execution (top-1 each), while Function memory is retrieved reactively and injected only after a tool-calling error occurs. AMD is evaluated on four student models (Qwen3-4B, Qwen3-8B, Gemma4-E4B, Llama3.1-8B) across AppWorld, BFCL V3, and ToolSandbox, against zero-shot and three adapted memory baselines (ReasoningBank, MemP, SASM).

Key results:

• Average accuracy gains over zero-shot across four students: +27.2 points on AppWorld, +11.2 points on BFCL V3, and +3.4 points on ToolSandbox.
• On AppWorld, AMD lets students match/exceed the teacher (GPT-5-mini, 50.00%): Gemma4-E4B reaches 54.17%, Qwen3-8B reaches 51.79%, Qwen3-4B reaches 49.40% (vs. 14.88% zero-shot).
• On BFCL V3, Gemma4-E4B (40.63%) and Qwen3-8B (40.96%) surpass the teacher's average (38.39%).
• Ablation: Subtask memory gives the largest incremental gain (e.g., +25.0 points for Qwen3-4B over Workflow-only on AppWorld); a "Student Memory" variant (self-generated instead of teacher) performs close to zero-shot, confirming teacher trajectory quality is essential.
• Representation matters: natural-language works best for Workflow memory (49.40% vs. 44.05% code-centric) while code-centric works best for Subtask memory (49.40% vs. 23.21% natural language); using natural language for all three types drops accuracy to 26.19%.
• Retrieval count k=1 is near-optimal; increasing k degrades performance, most sharply for Subtask memory (49.40% to 33.34% as k rises from 1 to 5).
• Teacher effectiveness is not monotonic with teacher accuracy: for Qwen3-4B, GPT-5-mini (50.00% teacher accuracy) outperforms as a teacher (49.40% student accuracy) versus stronger teachers like DeepSeek V4 Pro (81.55% teacher accuracy, only 38.10% student accuracy), indicating teacher-student compatibility matters beyond raw teacher capability.
• Accuracy gain from AMD peaks at the 4B student scale (+34.52 points for Qwen3-4B on AppWorld) and diminishes at 8B/14B, where students already approach teacher-level performance.

Why it matters / caveats: AMD offers a training-free way to substantially boost small tool-use agents by structuring teacher knowledge at matched granularities rather than dumping raw trajectories, and shows students can even surpass their teacher. Limitations acknowledged by the authors: evaluation is limited to text-based, structured tool-use benchmarks (not multimodal or open-ended coding settings); memory is built offline from a fixed teacher trajectory set and frozen at inference (no online adaptation to the student's own experience or distribution shift); and results depend on teacher-student compatibility, with adaptive teacher selection left as future work.

Problem: How to scale a sparse Mixture-of-Experts language model to hundreds of billions of parameters while keeping training numerically stable, communication/memory-efficient at long context, and capable of consolidating multiple specialized skills (agentic, coding, reasoning, long-context, abstention) into one deployable model.

Method: Motif 3 is a 314B-total / 13.2B-activated-parameter decoder-only MoE (53 layers, 384 routed experts + 1 shared expert, top-8 routing) built around Grouped Differential Latent Attention (GDLA), which fuses Grouped Differential Attention with Multi-head Latent Attention's compressed KV latent and adds a query-dependent output gate, using a hybrid schedule of 1 full-attention layer per 3 sliding-window layers (window 128). It further uses modified manifold-constrained hyper-connections (mHC, with post-mapping scale annealed from 2 to 1), Expert-Specific PolyNorm expert activations, and DeepSeek-V3-style multi-token prediction. Pretraining uses ~12.5T tokens (a custom SuperBPE-based Motif tokenizer, ~70% from NVIDIA Nemotron pretraining releases plus in-house web/STEM/code/math/Korean data), a WSD learning-rate schedule, the Muon optimizer (with Parallel Muon for MoE experts and QK-Clip for logit control), MXFP8 low-precision compute/communication, and window-aware context parallelism to extend training to 256K tokens. Post-training combines general SFT, six domain-specialist teachers trained with GRPO (DAPO-style) across 13 verifier domains plus a software-engineering teacher trained via SFT, consolidated into the final student via Multi-teacher On-Policy Distillation (MOPD) with ICE-POP importance-weight filtering.

Key results:

• GDLA reaches a training loss of 3.2 using 9.2% fewer training tokens than plain MLA in controlled ~10B-parameter comparisons.
• On agentic/tool benchmarks: 94.7 on τ2-Bench Telecom, 35.3 on τ3-Banking (highest among compared models), 74.9 on Terminal-Bench 2.1, 38.7 on GDPval-AA v2, 51.5 on the public subset of ITBench-AA (highest listed).
• Coding/reasoning: 76.2 on SWE-bench Verified, 40.6 on SciCode, 83.2 on IMO-AnswerBench, 75.5 on Apex Shortlist, 83.4 on GPQA Diamond, 37.0 on HLE.
• AA-Omniscience: 30.1% accuracy but 71.6 non-hallucination score (among the highest of compared models: MiniMax-3 428B-A23B, GLM-5.1 744B-A40B, Kimi-K2.6 1T-A32B, Qwen-3.7 Max, DeepSeek-V4-Pro 1.6T-A49B).
• Long context/instruction following: 72.3 on AA-LCR, 78.2 on IFBench.
• Pretrained base model (before post-training): 86.20 MMLU, 68.56 MMLU-Pro, 93.93 GSM8K, 70.58 MATH, 73.70 HumanEval pass@1.

Why it matters / caveats: Despite having fewer total/activated parameters than several compared open-weight models (e.g., Kimi-K2.6 at 1T-A32B, DeepSeek-V4-Pro at 1.6T-A49B), Motif 3 is competitive or best-in-class on agentic and tool-use benchmarks, suggesting the architecture/distillation recipe (GDLA, mHC, MOPD) yields strong per-parameter efficiency. Weaknesses remain in scientific coding (SciCode) and specialized scientific reasoning (CritPt) relative to the strongest peers, and the model is text-only with no visual capability yet.

Problem: Existing video-generation benchmarks measure perceptual quality and generic physical commonsense but lack multidisciplinary, expert-authored tasks with reusable evaluation criteria that test whether generated videos are faithful to domain-specific scientific mechanisms rather than just visually plausible.

Method: The authors build Sci-VBench, 1,253 expert-authored and independently reviewed video-generation prompts spanning 60 subjects across four disciplines (Natural Science, Healthcare, Humanities & Social Sciences, Engineering), each paired with an expert-written evaluation specification consisting of a reference guide (target concept + phase-based expected phenomena) and a rubric with 1–5 scoring anchors across four dimensions: Low-level Perceptual Fidelity (LPF), Prompt Grounding (PG), Scientific and Causal Correctness (SCC), and Spatiotemporal Consistency (SC). 61 expert annotators (11 undergrad, 45 grad students, 5 authors) authored and cross-reviewed examples via a textbook-guided pipeline; evaluation combines VBench-based Vision Tools for LPF with a rubric-conditioned MLLM-as-Judge (primarily Qwen3.5-397B-A17B, three runs averaged) for the other three dimensions, validated against expert human ratings. 16 frontier text-to-video models (8 proprietary: Sora-2, Veo-3.1/Veo-3.1-Fast, Kling-2.6, Wan-2.6, Seedance-2.0, HappyHorse-1.1, Gemini-Omni-Flash; 8 open-source: HunyuanVideo-1.5, LTX-2/LTX-2.3, LongCat-Video, Wan2.2-5B, CogVideoX1.5-5B, Cosmos3-Nano, MiniMax-H3) are benchmarked on a 150-example testmini split (and the full 1,253 for open-source models).

Key results:

• Inter-expert reliability is high: Cohen's κ = 0.842 on re-rated reference labels; rubric-consistency audit gives quadratic weighted κ of 0.79 (PG), 0.75 (SCC), 0.73 (SC).
• Providing the evaluation specification raises non-expert–expert Pearson correlation from 74.0 to 82.5 (average across dimensions), with the largest gains on SCC (68.2→82.5) and PG (74.1→83.3); the best MLLM judge (Qwen3.5-397B-A17B) reaches only 63.4 average, still beating prior auto-eval methods (VideoScore 47.5, VideoScore2 51.6, VideoReward 47.4, ETVA 47.9).
• Automatic perceptual quality (VT) is nearly flat across all 16 models (3.79–4.12), while SCC spreads widely (automatic: 1.24–3.34; human: 1.12–3.06), showing reasoning dimensions—not visual quality—separate current systems.
• Gemini-Omni-Flash tops both automatic (3.38) and human (3.18) averages; best open-source model (MiniMax-H3) reaches only 1.63 on SCC, about half the proprietary best (3.34), while open-source models are competitive on SC (Wan2.2-5B best automatic SC score of 2.79).
• Prompt rewriting (via Gemini-3-Flash) improves SCC most (+23.3% Wan2.2-5B, +51.7% HunyuanVideo-1.5), PG next (+12.4%/+26.7%), but SC least (+7.5%/+16.5%), and the open/proprietary gap narrows without closing (rewritten HunyuanVideo-1.5 SCC=2.20, still below best proprietary 3.34 verbatim).

Why it matters / caveats: The results show visual realism in current video generators has not translated into reliable scientific/causal reasoning, especially for open-source models, and that rubric-conditioned, expert-specified evaluation can be made reproducible for non-experts and MLLM judges alike, offering a scalable path to benchmark future "world simulator" video models.

Problem: In conversational image-editing systems, follow-up edit suggestions must be grounded in the current image state, but existing query-suggestion and image-editing recommendation methods either ignore visual context or don't jointly optimize for user click preferences and visual validity.

Method: A three-stage framework built on Qwen3-VL-8B: Stage 1 fine-tunes the policy (rank-4 LoRA, frozen visual encoder) on SFT targets constructed from 100K real Qwen App conversations, using a human-reviewed table of 61 follow-up editing intents and Gemini 3 Flash as a teacher to generate six-suggestion candidate slates, validated into 41.6K full slates (39.5K training). Stage 2 trains an 8B vision-language reward model via Bradley-Terry loss on 173,071 position-aware click preference pairs (clicked suggestion vs. unclicked suggestions shown above it), then runs multi-objective GRPO with five normalized rewards (click preference, output-gate validity, PPL vs. SFT policy, content-aware length, within-slate diversity via Qwen3-Embedding) and dynamic reward weighting. Stage 3 adds a frozen Qwen3-VL-30B-A3B "image-first" verifier that splits each suggestion into required sources and target state, checks whether sources are visible and targets already satisfied, and feeds a sixth grounding reward into a separate GRPO run (verifier used only in training, not at serving time).

Key results:

• Audit of 100K real conversation turns: 80.1% of follow-up edit queries are image-dependent (vs. 19.9% text-dependent).
• Offline: visual inconsistency rises from 3.0% (SFT) to 3.7% (Stage 2 click-RL) but drops to 0.9% with the full framework (75.7% relative reduction vs. Stage 2); redundancy falls from 23.3% (PE baseline) to 8.8%; expert GSB rises from +332 (SFT) to +405 (Stage 2) to +446 (full framework).
• Stage 3 cuts missing-source errors by 40.0% (0.70%→0.42%) and already-satisfied-target errors by 84.0% (3.00%→0.48%) relative to Stage 2.
• Source-target structured verifier recalls 78.7% of visual inconsistencies with only 0.6% false rejection, vs. 47.5% recall/22.2% false rejection for a single-pass VLM baseline.
• Position-aware click pairing improves held-out click accuracy from 0.619 to 0.690 (matched 51.9K budget, 2B backbone); scaling RM to 8B improves expert agreement (0.569 vs. 0.490) but a persistent tier-0/tier-1 inversion (0.041) shows click supervision alone cannot certify visual validity.
• 14-day live A/B test (millions of users, 5% traffic per arm) vs. prompt-engineered baseline: +32.70% recommendation CTR, +16.32% image take-away rate, +39.90% average conversation turns per user (all p<0.05).

Why it matters / caveats: Demonstrates that behavioral (click) alignment and visual-consistency supervision are complementary and neither substitutes for the other—click-only RL actually worsens visual grounding despite improving expert-rated quality and CTR. The verifier is training-only, so the deployed system adds no serving latency, making the approach practical for production; however gains are validated on a single internal product (Qwen App) and rely on proprietary human-reviewed intent tables and large-scale click logs that may not transfer directly to other systems.

Problem: Standard reverse-KL on-policy distillation (OPD) is mode-seeking and can under-cover plausible alternative continuations, and teacher entropy alone (as used by prior work EOPD) cannot tell whether uncertainty is concentrated in a few good candidates the student lacks, nor whether high teacher probability actually predicts downstream success.

Method: SPOT is a three-stage acquisition–exploration–exploitation procedure added on top of OPD. Acquisition scores each token position as the product of normalized teacher entropy, the probability mass captured by the teacher's top-ks candidates, and a student–teacher mismatch term (mass under-coverage plus Jensen-Shannon shape divergence), then selects the top-M positions as a sparse probing budget. Exploration appends each of the teacher's top-kp candidates at selected positions, rolls out Np student continuations per candidate under the frozen behavior policy, and scores them with a verifier to estimate branch values; only positions with at least one positive-reward candidate are retained (B+). Exploitation solves a closed-form KL-regularized reward-tilting of the teacher distribution (Eq. 10) to build outcome-calibrated local targets, added as an auxiliary branch loss (weight β) to the standard OPD loss. Experiments use Qwen3-8B (thinking disabled) as teacher and Qwen3-0.6B/1.7B/4B-Base as students, trained on MATH (0.6B/1.7B) or DAPO-Math-14k (4B), and evaluated zero-shot (Avg@8/Pass@8, 8 samples, temp 1.0, top-p 0.8) on MATH-500, AIME 2024, AIME 2025, AMC 2023, Minerva Math, and HMMT 2025, against KD, plain OPD, GRPO, and EOPD baselines.

Key results:

• SPOT achieves the best macro Pass@8 at all three student scales, improving over OPD by 4.55–5.28 points and over EOPD by 2.49–3.19 points macro Pass@8.
• Macro Avg@8 improves over OPD by 0.47–1.48 points and over EOPD by 0.29–0.68 points, ranking best or second-best in all settings.
• On Qwen3-0.6B-Base, the full acquisition score (H·C·G) reaches 21.51 Avg@8 / 41.60 Pass@8 macro, versus 19.40/35.72 for entropy alone — an ablation confirming each component (top-k mass C, student-teacher gap G) contributes.
• Removing verifier-guided calibration (Qwen3-1.7B-Base) drops macro Avg@8/Pass@8 from 31.78/54.87 to 28.57/47.49 (a 3.21/7.38-point loss), with the largest single-benchmark Pass@8 gain of +13.33 on AIME24.
• At a larger evaluation sampling budget (k=64), SPOT retains 12.50–16.67-point Pass@k gains over OPD on AIME24/25 and AMC23, showing the coverage advantage persists beyond Pass@8.
• β=0.1 is the chosen branch-loss weight based on sensitivity sweeps over {0.05, 0.1, 0.5, 1.0}; results also generalize out-of-domain and to a Llama model family per the appendix.

Why it matters / caveats: SPOT shows that decoupling "where to probe" (uncertainty + student mismatch) from "what to distill" (verifier-calibrated targets) yields notably better multi-sample solution coverage (Pass@k) without sacrificing average accuracy, at a controlled rollout cost (M×kp×Np extra continuations per trajectory). Gains in average accuracy (Avg@8) are more modest and sometimes only second-best versus EOPD, indicating the method's main benefit is broader coverage of viable reasoning paths rather than uniformly higher per-sample accuracy.

Problem: In long-context, long-generation LLM decoding, the KV cache dominates HBM capacity and bandwidth, so existing sparse-attention, KV-retrieval, and KV-prefetch schemes either fail to free HBM capacity or place slow off-GPU KV transfers on the decode critical path, capping batch size and throughput.

Method: OasisKV, built on vLLM's V1 engine, keeps only a bounded per-head sparse KV working set (Quest-style block min/max summaries) in GPU HBM while the full KV cache lives in CPU DRAM (or remote memory under PD disaggregation). It uses draft tokens from an EAGLE-3 multi-token-prediction head (e.g., Tengyunw/qwen3_8b_eagle3, nvidia/Qwen3-235B-A22B-Eagle3, TanBaby/EAGLE3-LLaMA3.1-Instruct-8B-YARN-64K) as a training-free lookahead signal, forking a sync-free background attention path each layer to predict the next step's top-K blocks and prefetch them asynchronously over PCIe via a persistent, layer-pipelined C++/CUDA worker chain (top-K prediction → KV selection → KV transfer), bounded by a capped-eviction policy to stay within per-step PCIe budget. A head-wise logical-to-logical mapping layer sits above vLLM's PagedAttention tables to support per-head sparse block sets, and "remote partial fetching" extends the same lookahead mechanism across the network for PD-disaggregated serving, transferring only a predicted KV union at admission plus on-demand misses during decode instead of the full KV cache.

Key results:

• Draft-token lookahead predicts the true next-step top-20 KV blocks with 98.74% average agreement across layers (vs. 83.9% for a previous-token proxy).
• Accuracy stays within 0.7 points of full attention under a 2,048-token KV budget (e.g., LongBench v2 overall −0.4 to −0.6 points; AIME24/AIME25/GPQA reasoning pass@8 within −0.66 points).
• Up to 2.1× decode throughput over dense vLLM on Qwen3-8B (single H100, 16K context, max concurrency 128: 1,398 vs. 676 tok/s), and up to 1.9× on Qwen3-235B-A22B under TP8.
• 1.69× throughput on a real AIME24 reasoning workload (2,083 vs. 1,235 tok/s on Qwen3-8B) at only 0.1 accuracy points lost; 1.20× on Qwen3-235B.
• Under PD disaggregation, 2.1–2.3× dense throughput while cutting per-request decode-node KV data by 6.5–9.7× at admission and reducing peak host DRAM occupancy by 2.2–2.6×; estimated TTFT reduction up to 2.2× at 90% prefix-cache hit rate over a 100 Gbps link.
• Outperforms KV-prefetch baselines ShadowKV, InfiniGen, and FreeKV across all tested concurrency/context settings, including cases where those baselines OOM or are unsupported.

Why it matters / caveats: By decoupling full KV storage from HBM and using speculative-decoding draft tokens (already common in production serving) as a free, accurate prefetch signal, OasisKV turns decode-time sparsity into real throughput and larger batch sizes without a dedicated trained predictor. Caveats: gains are workload-dependent (smaller for MoE models like Qwen3-235B where KV is a smaller share of memory), the prototype forces rejection of draft tokens rather than combining prefetching with actual speculative decoding, and PD-disaggregation prefix-caching benefits are analytically estimated rather than measured end-to-end.

Problem: Trajectory-indexed utility learning for self-evolving LLM agent memory dilutes limited feedback over an ever-expanding state space, and because trajectory-level rewards are jointly assigned to co-retrieved memories, broader exploration to fix this ("memory-reward trap," MRT) ends up giving misleading positive updates to irrelevant/noisy memories.

Method: RoMeRL (Reduced-Order Memory Reinforcement Learning) replaces the growing per-trajectory utility state with a fixed 2×2 (4-coordinate) per-task state factorized by outcome polarity (positive/negative) and memory dynamics (consolidated/adaptive): Positive Consolidated Coordinate (most efficient successful trajectory), Positive Adaptive Coordinate (earliest success after a failure), Negative Consolidated Coordinate (highest-utility failure above a threshold), and Negative Adaptive Coordinate (most recent failure). Each coordinate stores one representative trajectory plus a Q-value updated via a standard exponential-moving-average rule (Q ← Q + α(r−Q)) on retrieval, with new representatives warm-starting from the inherited coordinate utility; retrieval ranks the ≤4 active coordinates by a weighted mix of embedding similarity and Q-value. The paper also provides theoretical analysis (gap-variance decomposition, feedback-concentration and erroneous-coordinate-occupancy theorems) motivating the reduction. Evaluated with frozen LLM backbones on LifelongAgentBench (OS, DB tasks), ALFWorld (6 task types), and AppWorld, against baselines RAG, Mem0, MemP, Pass@k, and MemRL (the closest prior runtime-RL memory method).

Key results:

• Highest overall macro-average score across 10 evaluation units: 0.753 vs. 0.724 for MemRL (+2.9 points).
• On AppWorld, improves Scenario Goal Completion (SGC) from 0.286 to 0.326 (+4.0 points) while staying competitive on Task Goal Completion (0.306 vs. 0.313).
• Reduces Cold-Q ratio (fraction of coordinates never updated post-admission) by 80.0%, from ~44.9% (MemRL) to 9.0%, and increases feedback density ~6.0× (4.96 → 29.93 updates per active coordinate) on the OS task.
• Cuts maintained memory-pool size by 84.4% (45K → 7K) and average LLM calls by 21.1% (570K → 450K) relative to MemRL.
• MRT stress test (10% noisy entries injected): RoMeRL limits positive noise updates to 2.4 and final noise ratio to 0.15%, vs. MemRL+UCB's 7.2 updates / 1.20% ratio, while achieving the highest round-10 success rate (82.0% vs. 79.2%/78.4%).
• Cross-model memory transfer: frozen memory state improves scores and reduces average steps across four backbone/task combinations (e.g., GPT-5.4-mini on OS: 67.0→81.6 score, 3.23→2.22 steps).
• Ablations show Negative Consolidated Coordinate (27.14% of occupied coordinates on OS, 42.78% on DB) is critical for both SR and cumulative success rate, while Positive Adaptive Coordinate (8.21%/5.05% occupancy) mainly boosts last-epoch success rate.

Why it matters / caveats: Shows that a bounded, semantically factorized memory state can outperform continually-growing trajectory-indexed memory while being far cheaper and more robust to reward misattribution, without any LLM fine-tuning. The authors note it still relies on outcome-level rewards (doesn't fully solve causal credit assignment), and estimating the theoretical transition parameters (γ, λ) governing erroneous-coordinate occupancy would require counterfactual rollouts not used in the practical implementation.

Problem: Vision-language models often produce correct-looking answers that are driven by language priors or irrelevant image context rather than by the specific visual evidence that should support them, and existing perception-aware RL post-training methods (global image perturbations, attention proxies) cannot test whether a sampled answer causally depends on that local evidence.

Method: The paper proposes Counterfactual Evidence Disentanglement (CED), a training-time audit that, for each sampled response, neutralizes an object-centric Evidence Region (via feature-space mean-token replacement, using weak COCO object-box proposals) and compares the resulting log-likelihood drop against matched non-evidence Regions to compute a bounded evidence margin m (Eq. 3). This margin is combined with an answer-correctness reward inside GRPO via a gating function g(m) to form the full training pipeline, Evidence-RL, using Qwen2.5-VL-7B as the primary base model (with cross-backbone validation on Qwen2.5-VL-3B, Qwen3-VL-8B, and Qwen3.5-9B) and a training set of 15,314 samples derived from COCO val2017 (attribute, counting, spatial task families). Evaluation spans nine public benchmarks (CountBench, SpatialEval, HallusionBench, VLMsAreBlind, FREAK, MathVista, MMBench, MMMU, ScienceQA) with no image overlap with training data.

Key results:

• On Qwen2.5-VL-7B, Evidence-RL achieves the highest average score (67.13 vs. 61.19 base), a +5.94 average absolute gain, and is the only RL method with non-negative deltas on all nine benchmarks (largest gains: +14.22 FREAK, +7.58 VLMsAreBlind, +6.84 MMMU).
• Under matched compute/data on Qwen3.5-9B (Table 3), Answer-CED improves all nine benchmarks over correctness-only reward (+12.58 average) and beats matched retrained VPPO by +13.12; retrained PAPO becomes unstable (25% repeated-segment rate, 58% budget exhaustion at 24k tokens).
• Cross-backbone: positive mean improvement on every backbone, non-negative gains on all 36 benchmark–backbone cells (e.g., +11.34 avg. on Qwen3.5-9B, +1.33 on Qwen3-VL-8B).
• Signal validation: on counting tasks, 99.5% of GRPO groups show non-constant rewards and 90.0% contain same-answer trajectories with different rewards, confirming discrimination beyond string matching.
• Proposal-quality ablation: replacing COCO proposals with random object boxes collapses the evidence margin from 0.268 to 0.015 and relevant-beats-random rate from 0.636 to 0.487 (chance level); IoU-graded degradation shows margin falling from 0.501 (high IoU) to 0.219 (low IoU/near random floor 0.147).
• Intervention-type comparison: mean replacement gives strongest discrimination (AUC 0.669) vs. zero replacement (0.641) and Gaussian noise (0.629); global random masking performs below chance (0.490).
• No text-only degradation: on eight text-only benchmarks, mean accuracy change is only −0.27 pp (max single-benchmark drop 1.5 pp).

Why it matters / caveats: The method targets a real gap — coarse image-sensitivity rewards can't distinguish genuinely grounded answers from shortcut ones reaching the same output — and demonstrates consistent transferable gains across four backbones without extra inference cost. Caveats: evidence proposals rely on COCO object annotations (weak, not question-specific), the CoT-variant of CED is prone to reward hacking via chain truncation (mean chain length collapsing to 3.6 tokens), and gains on some benchmarks (e.g., ScienceQA) are partly offset by CED correctly declining to answer blank-image items that prior-following models get "right" by guessing.

Problem: Interpretability is usually bolted on after training via post-hoc methods (probes, attributions, sparse autoencoders) that lack guarantees of faithfulness, and it is generally assumed that constraining a model toward human-meaningful structure sacrifices capability.

Method: The authors build an "inherent interpretability" training recipe that bakes interpretability constraints into data, architecture, objective, and losses rather than explaining a model after the fact. They construct Atlas, a pipeline that extracts hundreds of millions of free-form tags from millions of documents, canonicalizes them into a fixed library of over 33,000 human-understandable concepts, and trains an annotator that labels more than 1 trillion tokens of pretraining data (web text, code, math, academic prose) at the chunk level. The architecture, Causal Diffusion, uses a block-causal attention backbone (diffusion-style parallelism within blocks, autoregressive-style KV caching across blocks) with an additive concept bottleneck inserted before the LM head, making the logit decomposition into concept contributions algebraically exact; the masking objective gives a trained "absent" baseline for feature-removal attribution. They run IsoFLOP scaling sweeps across three orders of magnitude of compute on four model families (AR, AR+Concept, causal diffusion (CDLM), CDLM+Concept) and fit joint Chinchilla-style scaling laws to both validation loss and four interpretability metrics (Concept Loss, Concept Independence Loss, Concept Contribution, Known Concept Alignment). They then scale the recipe to Steerling-8B, an 8B-parameter causal-diffusion model pretrained on 1.2T tokens plus 150B midtraining tokens on a code/math-augmented mixture, and evaluate it against open peer base models on standard benchmarks.

Key results:

• Adding the concept module shifts compute-optimal scaling exponents by only a small, fixed per-backbone offset rather than a growing penalty; irreducible loss L∞ actually shifts downward with concepts added (AR: 1.857→1.193; CDLM: 2.658→1.942).
• Steerling-8B's validation loss (L*=2.72) is predicted within 0.11 nats by the joint Chinchilla fit extrapolated from small-scale runs, versus a 0.47-nat miss for a naive log-linear extrapolation (~4x worse).
• All four interpretability metrics improve with compute on both backbones (Concept Loss and Concept Independence Loss decrease; Concept Contribution and Known Concept Alignment increase), with R² between 0.49–0.75 for compute-only fits and up to 0.62–0.75 for joint parameter/token fits.
• The concept bottleneck adds only 4% of parameters at 8B scale, and by construction would fall below 1% at frontier scale.
• Midtraining improves every steering metric (mean concept 1.072→1.247, mean quality 0.972→1.064, harmonic mean 1.020→1.148) and cuts concept independence loss by 19%.
• On seven benchmarks (HellaSwag, WinoGrande, PIQA, MMLU, ARC-C, GSM8K, Math), Steerling-8B averages 51.6% versus peers such as LLaMA2-7B (48.2%), DeepSeek-7B (49.3%), Gemma1-7B (60.4%), LLaDA-8B (61.8%), LLaMA3-8B (60.0%), and OLMo2-7B (67.3%) — each peer trained on roughly 2–16x more compute — landing within approximately 10% of the peer average.

Why it matters / caveats: The paper argues interpretability can be a designed-in training objective that scales favorably with compute rather than a tax that worsens with scale, potentially enabling frontier-scale models with built-in, faithful attribution and concept-level steering without retraining. Caveats: Steerling-8B still trails larger/better-trained peers on several benchmarks (e.g., MMLU, GSM8K vs. Gemma/LLaMA3/OLMo2), the fixed 33,000-concept library and chunk-level supervision are early design choices the authors flag as unsettled, and interpretability-metric scaling fits explain only a modest fraction of per-checkpoint variance (R² 0.49–0.75).

Problem: Proprietary LLM APIs (Anthropic, OpenAI, Google) hide chain-of-thought reasoning behind encrypted/signed client-stored blocks, but these blocks turn out to be portable across sessions, users, and even models, letting attackers decrypt them.

Method: The authors show encrypted reasoning envelopes (AEAD blocks with signature/thinkingSignature fields) are encrypted under what appears to be a single global provider key, making them replayable across sessions, users, and sibling models. Their extraction attack captures an encrypted reasoning block from a heavily safeguarded frontier model (e.g., Claude Opus 4.8, GPT-5.6 Sol, Gemini 3.1 Pro) and injects it into a weaker, less-safeguarded sibling model from the same provider (Haiku 4.5, GPT-5.6 Luna, Gemini Robotics 1.6), coercing that weaker model with a simple ad-hoc jailbreak prompt to transcribe the trace verbatim; fidelity is checked by comparing API-reported thinking-token counts against re-encoded extracted-trace token counts on 120 Codeforces problems, AIME 2025, and Humanity's Last Exam. They further scrape 6,708 public agent trajectories from GitHub/Hugging Face to decode 315,320 reasoning blocks for a secret-extraction study, and demonstrate prompt-injection attacks by planting malicious instructions inside encrypted reasoning blocks (e.g., in Claude Code / PostTrainBench-style long-horizon agentic traces).

Key results:

• Of 315,320 decoded reasoning blocks scraped from 6,708 public sessions, 0.3% (1,028 blocks) contained a privacy leak; 4.9% (328) of the 6,708 sessions leaked at least one sensitive item.
• Recovered 367 PII artifacts and 182 credentials overall (912 total artifacts including benchmark-sourced ones); genuine user sessions alone yielded 62 API keys, 33 passwords, 24 access tokens, 7 private keys, 30 personal emails, 6 non-localhost IPs, 130 names, and 36 postal addresses.
• 64 of 704 artifacts recovered from genuine-user reasoning traces were entirely absent from the visible chat history (silently leaked via hidden reasoning).
• Decoding Opus 4.8 reasoning through Haiku 4.5 reveals harmful information (e.g., car-theft methods) that was withheld from the model's benign final answer (proof-of-concept using a paraphrased HarmBench prompt).
• Estimated cost to decode a 10k-trace corpus (12k-token windows) via Claude Haiku 4.5 pricing: ~$720.
• Prior answer-only trace-inversion distillation raised a fine-tuned Qwen2.5-7B-Instruct's MATH500 accuracy from 68.4% to 76.0% versus answer-only distillation (cited prior work), whereas this paper's method recovers genuine verbatim reasoning rather than a surrogate reconstruction.
• Demonstrated a working prompt-injection chain where an injected encrypted reasoning trace causes GPT 5.6 Sol / Opus 4.7 (via Claude Code scaffold) to exfiltrate files to an attacker-controlled server without any plaintext trace visible to monitors.

Why it matters / caveats: The vulnerability defeats providers' anti-distillation and safety mechanisms simultaneously by turning a weaker sibling model into a "decryption oracle," and it creates a real-world privacy risk since developers unknowingly publish session logs containing encrypted secrets they cannot inspect or sanitize. The authors disclosed the issue responsibly (building on a prior May 2026 disclosure by Green) and all tested extraction attacks stopped working after providers were notified; findings are limited to the specific API/model versions available circa July 2026 and to a non-exhaustive scan of public traces, so real-world exposure (especially in private/production traces) is likely larger.

Problem: Existing evaluations cannot isolate an LLM's intrinsic ability to autonomously improve its own agent harness (the code that orchestrates reasoning, tool use, and memory) from base model strength, task-specific overfitting, or short-horizon effects.

Method: Evo-Bench fixes a policy model (DeepSeek-V4-Flash) and lets an "evolver" LLM iteratively edit a minimal CodeAct seed harness over a fixed budget (20 iterations / 1,000 steps / 48 hours), evaluating on a visible 160-task validation suite before final scoring on a disjoint 448-task held-out suite spanning Search (BrowseComp, HLE), Office (GDPval, APEX-Agents), and General agent tasks (Claw-Eval). The benchmark is built via a two-stage harness-guided construction framework: Stage 1 runs controlled evolution on 320 auxiliary tasks with four frontier evolvers (GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, GPT-5.6-Sol) to produce 12 diverse representative harnesses (Haux); Stage 2 scores 2,329 candidate tasks under these 12 harnesses to compute per-task "harness sensitivity" (Pearson correlation between task score and leave-one-out harness quality) and difficulty, filtering out insensitive tasks and stratified-splitting the rest into validation/evaluation suites. Metrics are Overall Score (final held-out performance) and Anytime Validation Score (average best-so-far validation score across the budget).

Key results:

• Top evolvers GPT-5.6-Sol and Claude Opus-4.8 achieve Overall scores of 46.3 and 45.8, gains of +16.6 and +16.1 points over the CodeAct seed baseline (29.7), approaching the human-engineered Artificial harness composite score of 47.5.
• Domain-dependent gains: Search sees massive improvement (Claude Opus-4.8 +34.8, GPT-5.6-Sol +32.8, nearly matching Artificial harness's 46.7); General tasks are where evolvers (GPT-5.6-Sol 59.4, Qwen3.7-Max 59.4) surpass the Artificial harness (56.3); Office tasks lag most, with several models showing near-zero or negative gains (e.g., Qwen3.7-Max -0.6) against the 43.9 human baseline.
• Evolution shows early saturation: Claude Opus-4.8 and GLM-5.2 post the highest Anytime Validation scores (51.4, 51.0) despite not topping the final Overall ranking, indicating rapid early gains followed by detrimental later edits.
• Cost varies widely: GPT-5.6-Sol exceeds $500/run for the top score, while GLM-5.2 and Qwen3.7-Max deliver strong performance for under $40, and DeepSeek-V4-Pro produces functional improvements for under $1.
• Robust cross-policy transfer: swapping the policy model to Qwen3.6-35B-A3B or GLM-5.2 still yields large gains (e.g., Qwen3.6-35B-A3B baseline 13.9 → 27.9–29.2 after evolution; GLM-5.2 baseline 38.0 → 48.4).
• Budget scaling: expanding budget from 24h/10 iterations to 48h/20 iterations produces monotonic Overall/AnytimeVal improvements for both Qwen3.7-Max and GLM-5.2.

Why it matters / caveats: Evo-Bench provides the first controlled, sensitivity-calibrated benchmark isolating genuine harness-evolving capability from model strength or overfitting, showing frontier LLMs can already rival or beat human-engineered agent harnesses in some domains (General, Search) but still struggle with highly specific Office workflows; the authors note current evolvers react superficially to aggregate scores rather than diagnosing causal failure modes and underutilize their research budgets.

Problem: Existing general-purpose robotic reward/value models rely on task-internal anchors like preference pairs or normalized [0,1] progress, which don't transfer cleanly across embodiments, viewpoints, and heterogeneous data sources.

Method: RynnValue is a language-conditioned value foundation model built on RynnBrain (8B and 4B variants) that replaces preference/progress supervision with temporal distance — the directed cost-to-go (in seconds) from an observation to a language-specified goal, discretized into 256 symlog-spaced bins and predicted via two-hot distributional classification. The model uses grouped repeated query tokens (N=8) for absolute-value and relative-value predictions per observation (K=8 sampled frames), with value-isolation attention preventing value queries from attending to each other or to context, plus random temporal sampling, temporal-order shuffling, and instruction-mismatch augmentation (10% of samples) to suppress shortcut learning. Training data comes from over 7,000 hours / ~3M instruction-conditioned clips (from 1.67M original episodes across AgiBot, EgoDex, Galaxea, InternData-A1, Open X-Embodiment, RDT, RoboCOIN, RoboMIND, RoboTwin, Soft-FOLD) with labels derived directly from timestamps via subtask segmentation and cutoff relabeling (no preference pairs needed); joint training combines absolute/relative temporal-distance cross-entropy losses with a language loss (video description, instruction-matching, success prediction) via AdamW (lr 1e-6). The learned value is converted into a reward via potential-based shaping (Φ = -v) for downstream RL.

Key results:

• RynnValue-8B achieves average Kendall's τa of 0.675 on RBM-EVAL-OOD (6 OOD datasets), beating the fully preference-supervised Robometer (0.655) and more than doubling a progress-only ablation (0.292); RynnValue-4B reaches 0.670.
• Among preference-free methods, RynnValue improves the strongest prior average from 0.502 to 0.670–0.675.
• Instruction-trajectory alignment: RynnValue achieves the highest normalized diagonal margin of 0.79 vs. 0.67 for the strongest baseline.
• Ablations: removing temporal-order shuffling drops average τa to 0.189 (largest degradation); uniform (vs. random) sampling drops it to 0.379; removing value-isolation attention drops it to 0.482; removing language supervision drops it to 0.537; removing relative-value supervision drops it to 0.627.
• Scaling analysis: task diversity reduces error monotonically with more training tasks, while episode-volume scaling (fixed tasks) saturates almost immediately.
• Real-world RL (dual-arm Franka, 4 manipulation tasks, potential-based shaping reward): online RL success rises from 52.5% (Robometer) / 48.8% (sparse) to 72.5% with RynnValue; offline RL (IQL on mixed-expertise data) rises from 63.8% (Robometer) / 23.8% (SFT) to 82.5% with RynnValue, while also using fewer average action chunks (e.g., 100% success with 16.8 chunks on Bread Basket Placement vs. 80%/18.9 chunks for Robometer).

Why it matters / caveats: Shows that a scalable, label-cheap supervision signal (timestamps only, no preference annotation or task-specific progress normalization) can match or beat preference-supervised state-of-the-art value models and translate into real robot policy gains via reward shaping. Caveat: gains are limited in visually ambiguous, precision-sensitive tasks (e.g., Box-in-Drawer Placement), where third-person RGB alone can't disambiguate grasp/placement quality, so the reward model gives only marginal online improvement there.

Problem: Existing agent-evaluation tooling (e.g., Inspect AI, OpenInference/Phoenix) only partially covers the workflow of benchmark orchestration, harness-agnostic trajectory capture, and multidimensional evaluation, forcing costly per-harness adapter engineering and leaving harness-level differences hidden behind outcome-only accuracy scores.

Method: A2E is a three-layer engine: a Task Layer built on the proposed Agent Task Protocol (ATP), which decouples benchmarks from agent harnesses via TaskInput/AgentBinding/AgentRunner/TaskTrace objects so any of 23 benchmarks can be paired with any of 9 agent harnesses (Agno, AutoGen AgentChat, CrewAI, Google ADK, LangGraph, LlamaIndex, OpenAI Agents SDK, Smolagents, Anthropic/Claude Agent SDK) without pairwise integration code; a Monitor Layer that instruments execution using OpenInference/OpenTelemetry-compatible spans to capture model calls, tool calls, latency, tokens, and errors as structured traces; and an Evaluation Layer performing "Lifecycle-Aligned Evaluation," registering rule-based metrics (accuracy, success rate, latency, cost) and LLM-as-judge metrics (reasoning, tool-use quality, safety) under four stages — Reasoning, Action, Final Answer, Runtime Quality — all persisted in a database for incremental re-evaluation. Experiments hold the backbone model fixed (DeepSeek-V4-Pro, FP4, 5 sampled tasks per harness–benchmark cell, 1035 scored runs) to isolate harness effects, and a second study uses GLM-5.2 across GDPVal, MMLU-Pro, and τ3-bench with a token-vs-success-rate trade-off score.

Key results:

• Across the 23-benchmark x 9-harness grid (1035 runs), average correctness ranges narrowly from 0.57 to 0.68, with Agno highest overall (0.68) but LlamaIndex leading on the 19 non-sandbox benchmarks alone (0.77 vs. 0.74 for Agno).
• Multi-turn benchmarks show much wider harness spread than single-turn ones: τ-bench 0.00–0.60, gdpval 0.00–0.60, traject-bench 0.20–1.00, while single-turn QA/coding tasks (arc-challenge, gsm8k, humaneval) score identically (1.00) across all 9 harnesses.
• On the 855 non-sandbox trajectory runs, mean token cost varies 3.5x across harnesses (2,063 tokens for Claude-Agent-SDK vs. 7,319 for Smolagents) while correctness spans only 0.568–0.663.
• With a fixed GLM-5.2 backbone, success-rate gaps across harnesses reach 0.20 on GDPVal, 0.30 on MMLU-Pro, and 0.66 on τ3-bench; no single harness ranks top-3 on all three benchmarks (winners: CrewAI/OpenAI Agents/AutoGen-AgentChat on GDPVal; OpenAI Agents/AutoGen-AgentChat/LangGraph on MMLU-Pro; LangGraph/Claude-Agent-SDK/Google ADK on τ3-bench).
• Case study on identical τ3-bench task: LangGraph succeeds (correctness=1.0) using 10,122 tokens/4 LLM calls/3 tool calls, while CrewAI fails (correctness=0.0) despite using 96,704 tokens (~9.6x more) and 9 LLM calls/5 tool calls.

Why it matters / caveats: The results show that outcome-only accuracy substantially underestimates differences among agent harnesses, and that harness effectiveness is model- and task-dependent rather than globally rankable, arguing for lifecycle-level, trace-based evaluation when choosing or co-developing harnesses. A caveat noted by the authors: cells are averaged over only 5 sampled tasks (score resolution 0.20), so per-cell variance is high and the correctness table is not meant as a definitive ranking.

Problem: LLM-based user simulators infer the next user turn's response and its underlying local interaction intent (e.g., accept vs. repair) jointly, so a fluent-looking continuation can advance the dialogue through the wrong intent without any explicit control over which plausible action is realized.

Method: UserIDA (User Intent-Directive Alignment) defines a six-way canonical intent taxonomy (Initiate, Amend, Supply, Repair, SetRegister, GroundAccept) and trains a LLaMA-3-8B-Base simulator in two stages: Intent-SFT (LoRA r=64) fine-tunes the model to generate the observed user turn conditioned on dialogue context, an implicit user profile, and an explicit per-turn intent directive, using turns from LMSYS-Chat-1M annotated by a frozen Qwen3.5-9B verifier (444,635 train / 27,778 val / 9,233 test non-ambiguous turns after filtering per the USP preprocessing protocol); then intent-calibrated policy optimization (a GRPO variant, LoRA r=16) samples K=4 candidates per prompt, scores them with a composite quality reward (semantic similarity via SimCSE, style similarity, AI-likeness penalty), and applies a group-specific calibration shift that forces every intent-violating candidate's reward below every intent-compliant candidate's reward (margin m=0.10) before standard group-relative advantage computation.

Key results:

• On LMSYS-USP turn-level test set, UserIDA reaches 86.6% intent accuracy and 0.864 macro-F1, beating the strongest dedicated simulator baseline (USP w/ Directive, 62.3%) by 24.3 points and the strongest general LM baseline (GPT-4o w/ Directive) by 9.3 points, while also achieving the best SimCSE (0.591), StyleCSE (0.476), Contextual Validity (86.7%), and User Authenticity (89.9%).
• Ablations: adding directives to prompting alone raises USP's accuracy only from 56.3% to 62.3%; Intent-SFT alone raises accuracy to 82.0%; quality-only GRPO further raises SimCSE (0.57→0.61) but drops intent accuracy to 80.7%, while full intent-calibrated RL recovers to 86.6% accuracy with 0.59 SimCSE.
• In controlled multi-turn trajectory evaluation, UserIDA improves step intent accuracy by 28.2 points and all-turn success by 45 points (13%→58%) over the strongest external baseline (USP w/ Directive).
• In within-context directive interventions (same dialogue state, six different requested intents), UserIDA realizes at least 4-of-6 target intents in 91.7% of contexts vs. 22.9% for the strongest external baseline, and raises intent-label coverage from 4.29 to 4.71 (out of 6) over the SFT-only variant.
• On rare intents, UserIDA raises Supply accuracy from 32.8% (USP w/ directive) to 73.4%, showing gains extend beyond the majority Initiate/Amend classes (81.4% of labeled turns).
• A retrospective Qwen3.5-9B intent verifier agrees with expert human labels at 87.9% accuracy (κ=0.822), while a prospective predictor without seeing the actual user turn reaches 71.4% accuracy (κ=0.595), showing next intent is only partially predictable from context.

Why it matters / caveats: The results support per-turn intent as a distinct, learnable control axis for user simulators, useful for reproducible counterfactual and multi-turn assistant evaluation (e.g., forcing repair vs. acceptance to probe clarification/error-recovery behavior). Caveats: the six-way taxonomy is a primary-action abstraction (multi-action turns collapse to one label via priority rules), annotation and reward signals rely on a single frozen LLM verifier (Qwen3.5-9B) rather than fully human-labeled data, and the prospective (autonomous) intent predictor's accuracy (71.4%) is well below the retrospective verifier's, indicating a meaningful gap remains for fully autonomous intent selection.

Problem: Standard retrieve-and-rerank fails when the input is only indirect evidence (e.g., a table cell or clinical mention) rather than an explicit query naming the target concept in a large taxonomy, leaving the gold concept reachable but ranked far from the top.

Method: The authors propose Factorized Hypothesis Search (FHS), which samples J partial "factorized hypotheses" (assignments over named semantic dimensions like FAMILY, ROLE, EVENT, QUALIFIER, SCOPE, TEMPORAL — 6 dimensions each for financial and clinical settings) from an unmodified LLM generator (Qwen3-32B) under stochastic decoding, renders each into both a label-style (programmatic) and definition-style (LLM) query, retrieves via BM25 (depth K=200) with a datatype pre-filter and label-coverage term, fuses rankings across hypotheses via Reciprocal Rank Fusion, and reranks the fused candidate pool with a candidate-level LLM verifier that issues per-dimension support/no-support/abstain verdicts. All compared methods (direct retrieval, HyDE/Query2Doc-style free-text grounding, self-consistency-style parallel sampling, decomposed retrieval, intrinsic/feedback iterative refinement, and FHS-Seq) share the same Qwen3-32B backbone, BM25 index, and a downstream Qwen3-32B listwise selector, and are evaluated on the FinTagging financial taxonomy benchmark (17,388 US-GAAP concepts, 2,509 test facts) and CodiEsp clinical diagnosis coding (71,344 ICD-10-CM codes, 3,144 test facts).

Key results:

• On Financial Tagging, FHS achieves Recall@1=0.185, MRR=0.257, and final accuracy=0.255, versus the strongest baseline's Recall@1=0.141 (feedback refinement), MRR=0.238, and accuracy=0.234.
• On CodiEsp, FHS improves Recall@1 from 0.201 (best baseline) to 0.264, MRR from 0.298 to 0.352, and final accuracy from 0.322 to 0.330, while also achieving the highest Recall@10 and Recall@50.
• Ablation: replacing the factorized hypothesis path with a same-size free-text ensemble drops Recall@1 from 0.185 to 0.116 and MRR from 0.257 to 0.194 (though Recall@50 rises from 0.543 to 0.583); removing the candidate-level verifier drops Recall@1 to 0.124 and MRR to 0.205 with Recall@50 unchanged.
• An oracle probe with gold concept label+definition as query reaches Recall@10=1.000, while raw-context direct retrieval reaches only Recall@10=0.240 (but Recall@200=0.752), showing the bottleneck is ranking position, not reachability.
• Sequential refinement (FHS-Seq) gives no improvement over FHS's parallel first round despite substantially greater inference cost (Recall@1 0.112 vs 0.185, accuracy 0.226 vs 0.255).

Why it matters / caveats: FHS offers a general, zero-shot recipe for grounding ambiguous evidence before taxonomy retrieval, useful for financial XBRL tagging, clinical coding, and similar large-inventory linking tasks, with clear evidence that structured multi-hypothesis search beats both single-pass grounding and costlier iterative refinement. Caveats: dimension schemas were hand-derived from existing taxonomy metadata (not automatically induced), and all LLM components use a single model (Qwen3-32B), so generalization across model families/scales is untested.

Problem: Existing egocentric data-collection hardware is either cheap but low-fidelity (monocular, rolling-shutter consumer cameras with no synchronized IMU, e.g. GoPro/Ego4D) or high-fidelity but closed and non-reproducible (Project Aria), leaving no open-source, hardware-synchronized stereo-inertial device for scaling egocentric capture across distributed contributor networks.

Method: Ego-OSCAR is a head-mounted capture rig combining a Dexcin hardware-synchronized global-shutter stereo USB camera (two Omnivision sensors, 1280x720 @30 FPS, 126° FOV, 42mm baseline), a Radxa Rock 5C (RK3588) SBC for hardware-accelerated MJPEG decode/H.264 encode, a TDK ICM-20948 6-axis IMU sampled at 120 Hz, and a Seeed Xiao ESP32-S3 microcontroller that bridges camera/IMU clock domains (via a Start-of-Exposure ISR plus an LED-flash anchor at the 60th interrupt) and runs a 1 Hz heartbeat watchdog; total BoM is ~USD 200 (INR 19,100), all parts COTS or 3D-printed. The team deployed 13 units over ~6 months across 25 contributors in India to collect the Ego-OSCAR-550h dataset (1,462 stereo sessions), annotated corpus-wide with free-form open-vocabulary action captions and per-frame 3D hand reconstructions (via WiLoR), plus per-session stereo calibration and synchronized IMU traces.

Key results:

• ~550 hours of stereo video per camera (~1,100 stereo camera-hours) across 1,462 sessions, 40+ indoor environments, 25 contributors, 13 devices.
• 209,315 free-form action segments covering ≈100% of the timeline (median 94 segments/session); 460 verbs, 32,630 object phrases, 57,104 verb-object combinations; top-20 expressions only 1.5% of instances (long tail).
• Per-camera reprojection error <0.03 px; mean epipolar error after rectification 0.4 px across all 13 devices; visual-inertial sync residual lag of 700 µs (Kalibr Cam-IMU test).
• IMU noise: accelerometer noise density 3.64e-2 m/s²/√Hz, gyroscope bias instability 9.68e-4 rad/s (consumer/phone-grade).
• WiLoR hand-detection rate of 94% across the full corpus; VINS-Fusion stereo-inertial VO converges on 12/20 held-out sequences (vs. 15/20 for an Intel RealSense with active stereo).
• 96% of deployment sessions produced usable data end-to-end; IMU present in 1,271/1,462 sessions (86.9%).

Why it matters / caveats: Provides a fully open, reproducible, sub-$200 stereo-inertial capture platform meant to lower the barrier for crowdsourced egocentric data collection at VLA/robot-learning scale, analogous to what ALOHA/UMI did for manipulation hardware. Caveats acknowledged by the authors: no policy-training experiments are run to show downstream value; no ground-truth pose (only VO convergence rate, no ATE/RPE) is reported; the corpus is geographically concentrated (India, domestic settings, 25 contributors/13 devices); and the consumer-grade IMU is the dominant pose-error source, swappable but not yet benchmarked.

Problem: Most vision-language grounding formulations assume the relevant text phrase or category is already specified and only localize it in the image, ignoring the more basic problem of first determining which parts of a text are visually referential at all.

Method: The authors formulate grounding as bidirectional concept correspondence: given an image-text pair, recover the full set of correspondences {(t_i, s_i)} between (possibly discontinuous/coreferential) text-span masks and instance-level image masks, without pre-specified text spans. They build ConCor-1 on a frozen-ish pretrained Qwen3.5 (0.8B) VLM backbone used with bidirectional attention as a joint image-text encoder, adding learnable "bridge tokens" (assigned during training via a multi-scale spatial grid + Hungarian matching) that attend over vision and text tokens and feed three lightweight heads: a text segmentation head (bilinear scoring), an image segmentation head (VLM vision tokens fused with vision-encoder features, conv upsampling, PointRend-style BCE+Dice loss), and a presence head (BCE). Training data (653K image-text pairs, 184K images, 3.1M correspondences, plus 614K images/7.8M masks of instance segmentation data) is built by repurposing GoldG, COCONut-PanCap, COCO/COCONut/EntitySeg/ADE20K, PixMo-Point, and a newly constructed GroundedRef-train, using an LLM rewriting pipeline to merge coreferent mentions into unified text masks; evaluation uses new correspondence-format validation sets on Flickr30k, COCONut-PanCap, GroundedRef, COCO, LVIS-minival, and EntitySeg, trained for 100K steps on 8 H100s.

Key results:

• On COCONut-PanCap, ConCor-1 improves JointF1 by 48% and mJS (joint IoU) by 41% over the strongest baseline (JointF1 88.8 vs. best baseline; mJS 89.5).
• On zero-shot LVIS-minival, ConCor-1 improves JointF1 by 29% relative over the strongest baseline (29.9 vs. 23.1), while processing the full category vocabulary in a single forward pass (baselines require chunking).
• Ablations: bidirectional attention over causal attention improves all metrics (e.g., JointF1 49.3→51.3); multi-scale spatial bridge-token assignment beats raster-order assignment (JointF1 52.4 vs. 43.4); fused VLM+vision-encoder features give best IoU metrics.
• Attention analysis: ConCor-1 achieves higher text-to-image attention IoU (up to 0.57 at layer 20) and top-1 hit rate (up to 0.75) with lower spatial entropy than base Qwen3.5 or Qwen3.5-FT across layers 4-20.
• Random-initialization ablation of ConCor-1 collapses performance (e.g., LVIS JointF1 drops to 0.1), showing pretrained VLM representations are essential.

Why it matters / caveats: The reformulation unifies phrase grounding, referring-expression grounding, and open-vocabulary detection/segmentation under one correspondence-prediction objective and could support new applications like evaluating caption groundedness or text-to-image faithfulness; however, it relies on an LLM-based rewriting/re-annotation pipeline (plus manual filtering) to convert existing datasets into the new format, and gains on smaller-vocabulary/simpler benchmarks (e.g., Flickr30k, COCO) are more modest than on long-caption or large-vocabulary settings.

Problem: Gradient descent on factored models W=UV⊤ is implicitly biased toward low-rank, minimum-nuclear-norm solutions, but it is unclear whether and why Adam-family optimizers preserve or destroy this bias, since the loss itself is invariant to an orthogonal "gauge" rotation (U,V)→(UQ,VQ) of the latent factors.

Method: The paper defines gauge-equivariance of an optimizer update rule under (U,V)→(UQ,VQ), proves gradient descent, Polyak/Nesterov momentum, scalar-preconditioned ("shared-scalar") Adam, Muon (matrix-sign update), and Shampoo (Kronecker preconditioner) are equivariant, while Adam, RMSProp, signSGD, Lion, and Adafactor are not (Propositions 4.2–4.4, structure Theorem 4.5, transfer Theorem 4.6). It then benchmarks nine update rules on an underdetermined 40×40 rank-3 matrix-sensing task (Gunasekar et al. setup, m=2·dof Gaussian measurements, small init 10⁻³, no weight decay, run to interpolation loss <10⁻⁷), sweeps a one-parameter "dial" p∈[0,1] interpolating Adam's coordinate-wise denominator to a shared scalar, trains twin 2-layer/4-head transformers on modular addition (mod 47) with gauge-rotated per-head query/key weights, and evaluates rank-48 factored matrix completion on Indian Pines and Pavia University hyperspectral image datasets at matched training loss.

Key results:

• On matrix sensing, equivariant methods reach recovery error ≤0.286 (Muon 6.8×10⁻⁶, GD 0.131, scalar-Adam 0.201, Shampoo 0.286) while coordinate-wise methods reach ≥0.42 (Lion 0.425, signum 0.445, RMSProp 0.527, Adafactor 0.543, Adam 0.573), a clean two-cluster split that holds up to n=256 over 10 seeds.
• Adam's per-head invariants W_Q⊤W_K diverge to 56% relative Frobenius distance between two function-identical (gauge-rotated) transformer twins, splitting already at step 1 (logit distance 3.6×10⁻³ vs. 2.9×10⁻⁷ for a 10⁻⁷-noise control, a 4-order-of-magnitude gap); SGD and scalar-Adam twins stay at floating-point noise level.
• On hyperspectral matrix completion at matched training loss, GD's held-out RMSE is 44% lower than Adam's on Indian Pines (0.0150 vs. 0.0268, at effective rank 11 vs. 28) at m/dof≈1.15, and the improvement is 43.8% on Pavia University at the same density, narrowing to ~28% (Indian Pines) and 22.9% (Pavia) at higher sampling density m/dof≈1.9, consistent across all 4 seeds.
• A phase-diagram over "tail energy" (fraction of target energy outside the low-rank part) locates a boundary near 4%, above which Muon's equal-rate spectral schedule stops helping and GD's greedy schedule wins.

Why it matters / caveats: The results reframe optimizer choice for factored/low-rank models (including attention Q/K weights and LoRA-style factorizations) as a decision about which interpolant is selected, not just a tuning detail — but the authors stress the bias is mild, helps only when the useful regularization itself is mild and aligned in direction, and equivariance alone (e.g. ScaledGD) is not sufficient for low-rank recovery since the spectral growth schedule also matters.

Problem: Existing agent benchmarks lack an end-to-end sandbox for verifiable multi-party, cross-user agent collaboration over personal (owner-scoped) workspaces that also tests how harmful actions and attacks propagate through the messages, tools, and approval paths of a human-centered agent network.

Method: WeClawArena models each human user as a workspace node (files, database, tools, policies, delegated agents) connected in a directed social/task graph, and provides a Docker-backed runtime sandbox with a message gateway that routes agent-to-agent communication, tool calls, and resource operations while passively recording all events for audit. The benchmark contains 124 human-annotated base tasks across six domains (bargaining, bidding, travel, SWE-Workspace, clinical, trading), each expanded into 620 scenario variants: one benign no-attacker control plus four attack-vector variants (collaboration, security, privacy, governance harm surfaces). Evaluation is run with the same Dockerized OpenClaw runtime across models, reporting task success rate (TSR) from final workspace state/evidence and a separate LLM-judged attack success rate (ASR, headline judge = GPT-5.2) computed only over attack-vector rows from bounded post-run evidence packets, with a Claude Opus 4.7 sensitivity pass for validation.

Key results:

• Benchmark scale: 124 base tasks, 620 scenario variants, 6 domains, 4 attack vectors + 1 benign control per base task.
• Overall TSR (all variants): Claude Opus 4.7 leads travel (83.0%), SWE-Workspace (34.0%), and bidding (55.0%); Claude Sonnet 4.5 leads bargaining (68.3%) and ties best clinical (40.0%). SWE-Workspace remains hardest overall (best model only 34.0%).
• Model-level macro-vector attack resistance (1 − ASR, ASR-MAIN-SIX-DOMAIN pool): Claude Opus 4.7 highest at 97.4%, followed by Claude Sonnet 4.5 (78.1%), Claude Opus 4.1 (74.2%), Kimi K2 Thinking (70.9%), Qwen3 32B (64.5%), Kimi K2.5 (64.0%), DeepSeek V3.2 (54.0%), Qwen3 235B (51.1%).
• Attack-vector-level resistance (1 − ASR): governance and security attacks are most successful (lowest resistance, 57.4% and 64.1% respectively), while collaboration attacks are least successful (highest resistance, 81.9%); privacy at 73.4%.
• Domain-attack interaction: security pressure dominates bargaining and bidding; privacy/governance pressure dominates SWE-Workspace; bidding shows near-zero governance harm — indicating harm surfaces cannot be collapsed into one "attacked" condition.

Why it matters / caveats: By keeping utility (TSR) and attack success (ASR) as separate, evidence-audited metrics over the same paired benign/attack tasks, the benchmark shows these are distinct axes — agents can complete tasks while leaking protected data or accepting invalid authority, or resist attacks while still losing task utility — which single-metric benchmarks would conflate. The study covers 8 LLM backbones on one fixed (OpenClaw) agent runtime, so results reflect that specific orchestration/tool-use harness rather than agent-framework-agnostic capability.

Problem: Existing out-of-context (OOC) benchmarks mainly check whether MLLMs refuse truly unanswerable questions but overlook "shifted in-context" (Shifted IC) questions that contain distracting or misleading context yet remain answerable, so models' balance between appropriate refusal and answer preservation is not well measured.

Method: MMOOC is a 41K-scale image-question benchmark spanning three question formats (yes/no, multiple-choice, open-ended VQA), five OOC categories (Multimodal Ambiguity, Visual False Premises, Uncertain Spatial & Physical Context, Unclear Logical & Symbolic, Missing Knowledge & Background) and three answerable Shifted IC categories (Misleading Premise, Partial Answerability, Image-Question Mismatch), across six visual scenarios/eight shift types. Data is generated via an in-context-learning pipeline using Qwen3.5-122B-A10B, GPT-4o, and o1 for grounded captions/questions/explanations, supplemented with manually designed questions and Auto-Shuffle-derived OOC samples from MME, MMStar, and OK-VQA (following UPD); quality is filtered by requiring agreement among GPT-4o, o1, and o3 on answerability, followed by human verification. Evaluation uses Accuracy and Answer Rationality for IC samples and Refusal Rate and Refusal Rationality for OOC samples, with rationality scored via an LLM-as-a-Judge protocol (GPT-5.6, Claude Opus 5, DeepSeek-V4-Pro) validated against human annotation. The paper evaluates 18 MLLMs (13 open-source, 5 closed-source) and tests post-training (SFT/DPO) and prompting (refusal prompt, CoT) mitigations.

Key results:

• Human-judge agreement was 96.78% for Refusal Rate and 89.34% for Rationality metrics, validating the LLM-as-a-Judge protocol.
• OOC performance is low and inconsistent: e.g. Qwen3-VL-2B scores only 5.75 (YesNo) and 8.25 (VQA) on the USPC category; o1 achieves the strongest overall OOC performance among evaluated models.
• Model scale does not monotonically improve OOC robustness (e.g., Qwen3.5-122B-A10B does not consistently beat smaller Qwen3-VL variants; Gemma-4 models lead open-source performance despite not being the largest).
• Models perform much better on Shifted IC than OOC tasks overall, but Partial Answerability in VQA remains weak even for closed-source models (e.g., Claude-Opus-4.6 scores only 32.25 on PA-VQA in Table 4).
• SFT post-training on Qwen3-VL-2B raises Refusal Rate from 8.00 to 38.00 and Refusal Rationality from 14.50 to 27.00, but general MMStar accuracy drops from 64.19 to 58.95, showing a refusal-alignment/general-capability trade-off.
• MMOOC contains the longest average response length (26.69) and lowest Self-BLEU (2.00) and semantic similarity (0.09) versus HaloQuest, MoHoBench, and UPD, indicating less templated data.

Why it matters / caveats: The benchmark highlights that stronger general reasoning (e.g., o1) does not guarantee stable contextual robustness, and that naive refusal prompting induces over-refusal on answerable IC questions, motivating alignment/prompting methods (e.g., CoT) that better balance answering and refusing. The work is limited to image-text interactions; extending to video, audio, and embodied settings is left to future work.

Problem: Existing agent architectures for embodied Intelligent Virtual Agents (IVAs) either offer low-level reactive control tightly coupled to game engines (state machines, behavior trees) or high-level cognitive reasoning models (BDI, SOAR, ACT-R) that are theoretically rich but difficult to implement in real-time 3D virtual environments, leaving a gap between cognitive reasoning and practical embodied deployment.

Method: CEAA (Cognitive Embodied Agents Architecture) is a three-layer, twelve-component modular architecture that extends the classical Sense-Think-Act paradigm with a blackboard-based shared Knowledge Base, explicit Memory/Memory Processor components, and a BDI-based Cognitive Construct (Beliefs, Desires, Intentions) reasoning model. The three layers are the User and Environment Layer, the Knowledge Layer, and the Agent Layer, comprising components: Environment, Knowledge Base (blackboard), Sense, Memory, Memory Processor, Think, Cognitive Construct, Reasoner, Planner, Behavior Mapper, Agent Capabilities, and Act. Feasibility was assessed via a prototype: four embodied pedagogical agents deployed in Unity3D within a virtual ENIAC/computing-history museum, built in two parallel versions (desktop and VR), evaluated through a mixed-methods comparative study.

Key results:

• Prototype evaluation involved 92 undergraduate Computing and Engineering students, randomly split 46/46 into desktop vs. VR conditions, each interacting for approximately 45 minutes with pre-test/post-test and post-experience questionnaires; 12 VR participants also joined focus groups.
• All participants showed statistically significant pre-test-to-post-test learning gains in both conditions; the desktop condition's aggregate gain was slightly higher than VR's, but the between-group difference was not statistically significant.
• Technology acceptance was positive in both conditions, with VR showing slightly higher median scores for effort expectancy and behavioral intention to use, while performance expectancy and attitude towards use were similar across conditions.
• No latency, throughput, or scalability benchmarks are reported — the paper explicitly states further technical evaluation is required.

Why it matters / caveats: CEAA is presented as a reusable implementation-oriented "brain" template bridging cognitive architectures and real-time engines (Unity/Unreal), but the authors state it remains conceptual/theoretical — validated only through one prototype application (a VR/desktop museum) and a learning-outcomes user study, not through architectural benchmarking against BDI, SOAR, ACT-R, or performance/scalability metrics; the authors flag this as future work, along with building Unity/Unreal extension tools.