AI papers — 2026-07-09

Problem: Applying AI to structure–property understanding across biology, chemistry and materials science requires models that both preserve domain-native 3D/topological structural information and expose explicit, inspectable reasoning linking that structure to predictions, a combination existing text-based LLMs and domain-specific predictors fail to jointly provide.

Method: SciReasoner is a multimodal foundation model (initialized from Qwen3) that discretizes protein structures (via Foldseek 3Di), small-molecule 3D conformers (via ConfSeq), and crystal structures (via SLICES) into a unified structure-aware token vocabulary, combined with sequence/text tokens in a single autoregressive LLM backbone. Pretraining uses a 3-stage continued pretraining pipeline (Stage 1 warm-up alignment of new structural tokens with frozen backbone, Stage 2 full-parameter multimodal training, Stage 3 annealed training with increased QA-style data), followed by a 2-stage post-training scheme called "self-bootstrapped native structural reasoning": Stage 4 intra-domain structural evidence grounding trains domain-specific experts (protein/molecule/material) via SFT+RL to use structural tokens as chain-of-thought evidence, and Stage 5 cross-domain reasoning consolidation pools expert-generated on-policy traces to train a single unified reasoner via SFT+RL.

Key results:

• Across 86 benchmarks spanning proteins, DNA, RNA, small molecules, and inorganic crystals, SciReasoner achieves state-of-the-art on 67 tasks and matches/exceeds published specialists on 26 benchmarks.
• Protein GO (CAFA-3/DeepFRI-GO): overall Fmax 0.59 vs BLAST 0.55, Foldseek 0.54, ESM2 0.53, SaProt 0.52, GPT-5.5 0.31; largest gain in low-homology Cellular Component bin (≤30% identity): Fmax 0.55 vs BLAST 0.34 (+0.21) and vs ESM2 0.42 (+0.13); CC overall 0.58 vs BLAST 0.49.
• Retrosynthesis (USPTO-50K, Exact Match): SciReasoner 0.72 vs RSGPT 0.63 (+0.09) vs Opus-4.7 five-shot 0.48; recovers correct Top-3 reactants in 5/5 representative cases vs 2/5 for RSGPT and 2/5 for Opus-4.7.
• 3D molecule similarity (DUD-E virtual screening, 102 targets): matches best-reported AUC of 0.76 and improves 5.0% enrichment factor from 7.12 to 7.70.
• Materials science: outperforms CGCNN on all 10 sub-tasks (Materials Project, JARVIS-DFT, SNUMAT, hMOF, QMOF) and exceeds LLM-Prop on most numerical properties; formation-energy parity R²=0.895, band-gap parity R²=0.785.
• Other domain results: subcellular localization accuracy 0.88 vs ESM2 0.84; RNA Isoform R² raised 0.59→0.86; RNA-protein interaction MCC 0.74→0.81; biomedical QA 0.85 BertScore; protein function description 0.77 ROUGE-L.
• Structure-ablation experiments show consistent performance drops when structural tokens are removed, especially for protein tasks.
• Double-blind human expert evaluation (N=1776 case-judgments) rated SciReasoner tie-or-better vs DeepSeek-V4-Pro in 98% of comparisons (73% strongly prefer, 21% prefer, 4% tie); average score 8.7/10 vs 4.3/10 across five rubric axes (evidence grounding, domain plausibility, target-regime alignment, reasoning coherence, anti-hallucination), all differences significant (Wilcoxon P<0.001).

Why it matters / caveats: By treating structural tokens (residues, molecular fragments, crystal connectivity motifs) as addressable, citable evidence within chain-of-thought rather than opaque inputs, SciReasoner links accurate specialist-level prediction to auditable, expert-verifiable scientific reasoning across biology, chemistry and materials; the human evaluation is described as a pilot study still collecting more judgments, and gains are attributed partly via attention/embedding analyses rather than fully controlled causal tests.

Problem: Vision-Language-Action models predict actions mainly from the current observation under a Markovian assumption, causing a temporal short-horizon bias that hurts long-horizon, temporally dependent manipulation tasks, and existing memory-augmented VLAs store history outside the model's native latent space, feeding it back only as auxiliary policy-side context rather than letting it participate in reasoning.

Method: LaMem-VLA is built on a 7B-parameter Prismatic VLM backbone (LLaMA-7B language model plus vision encoder) pretrained on Open-X Embodiment, paired with a ~300M-parameter diffusion-based action expert (following Diffusion Policy, trained with DDIM sampling, 10 denoising steps). Four coordinated modules close the loop: a latent memory curator maintains a short-term visual memory vault (key-value pairs from an SE-bottleneck compression module) and a long-term action-hidden-state vault (each capped at L=16 units, consolidated via cosine-similarity redundancy merging); a latent memory seeker builds a context-aware query from current visual/instruction tokens and retrieves the top-K=8 units from each vault by cosine similarity; a latent memory condenser reconstructs retrieved evidence into fixed-length latent tokens (Ls=8 short-term, Ll=4 long-term) via lightweight transformer "memory formers"; and a latent memory weaver prepends these tokens directly into the VLA input sequence (before action query tokens) so they undergo self-attention jointly with the current observation, instruction, and action queries before conditioning the diffusion action expert.

Key results:

• SimplerEnv-Bridge: 73.9% average success rate — beats baseline CogACT by 16.6 points, π0 by 4.7 points, and MemoryVLA by 2.0 points; per-task scores of 83.3% (Spoon on Towel), 75.0% (Carrot on Plate), 41.7% (Stack Cube), 95.8% (Eggplant in Basket).
• LIBERO (5 suites, Franka robot): 97.6% overall average — outperforms MemoryVLA by 1.1 points and CogACT by 4.4 points; first-four-suite average of 97.7% vs. π0's 94.2% (3.5-point gain); per-suite: 98.8% Spatial, 99.0% Object, 97.2% Goal, 95.8% Long-10, 97.0% Long-90 (2.4 and 1.4 points above MemoryVLA on the two long-horizon suites).
• Ablations: removing both memory streams drops performance to 57.3% (SimplerEnv) / 92.1% (LIBERO-90); removing only short-term or only long-term memory yields 65.6%/95.4% and 64.6%/94.8% respectively, showing complementary contributions.
• Latent-native vs. policy-side conditioning: baseline 57.3%/92.1%; adding memory as external policy-side context reaches 71.9%/94.8%; raw (uncompressed) retrieval reaches 69.8%/95.1%; full latent-native LaMem-VLA reaches the best 73.9%/97.0%.
• Retrieval budget K: performance rises from K=2 (66.7%/94.4%) to a peak at K=8 (73.9%/97.0%), then declines slightly at K=12 (71.8%/96.2%).

Why it matters / caveats: The results suggest that weaving compact short-term (visual) and long-term (semantic/action-continuity) memory tokens directly into the VLA's native latent reasoning sequence — rather than conditioning the policy externally — yields more robust, temporally aware manipulation without unbounded context growth. The authors explicitly note the current validation is simulation-only (SimplerEnv and LIBERO); real-world robot experiments are stated as future work for the next version.

Problem: Video generative models are optimized for visual fidelity and content creation rather than the physical realism, computational efficiency, and embodiment grounding required for robot control and embodied intelligence.

Method: LingBot-Video is a DiT-based, task-unified single-stream diffusion transformer (T2I/T2V/TI2V via 3D MM-RoPE, QK-Norm, adaLN-single modulation) that replaces dense FFNs with a sparse Mixture-of-Experts layer following DeepSeekMoE-style fine-grained expert segmentation, shared+routed experts, sigmoid group-limited routing, auxiliary-loss-free load balancing, and a sequence-wise balance loss; it scales from MoE 13B-A1.4B up to MoE 120B-A11B, uses Qwen3-VL-4B for conditioning and Wan2.1-VAE for latents, adds a rectified-flow cascaded refiner (480p→1080p), and is trained via a five-stage progressive pretraining curriculum on a data pipeline (Data Profiling Engine, World-Knowledge Topological Graph, Dense Structured Captioning, Caption Rewriter) that injects over 70,000 hours of robot manipulation/navigation/egocentric footage, followed by GRPO post-training with six specialized reward models (vision quality, text-video alignment, dynamic degree, motion coherence, human-motion consistency, physical plausibility) plus a negative-aware DiffusionNFT-style fine-tuning stage using real videos as preference pairs.

Key results:

• Expert scaling ablation: E=128 chosen as default (E=64→128 gives large gains, 128→256 marginal) at fixed 1.4B active params.
• Fine-grained routing (13B-A1.4B-E128, top-8/128) consistently beats coarser routing (13B-A1.5B-E64, top-4/64) despite the latter having more active FLOPs.
• MoE 13B-A1.4B and MoE 30B-A3B outperform dense models with ~2x their active parameters; MoE 30B-A3B approaches Dense 14B performance.
• Inference speed ratio at 1M-token sequences: MoE 30B-A3B is 0.97x vs Dense-3B (near parity), and 1.50x/2.59x/3.18x faster than Dense-6B/14B/30B respectively.
• RBench (robot-centric benchmark, 650 prompts): LingBot-Video scores 0.620 average, best among open-source models (Cosmos3 Super 0.581, Wan 2.2 A14B 0.507, HunyuanVideo 1.5 0.460, LongCat-Video 0.437) and above closed-source Wan 2.6 (0.607), Seedance 1.5 pro (0.584), Veo 3 (0.563).
• Physics-IQ Verified (I2V): LingBot-Video scores 40.4, ranking first among open-source models, ahead of Cosmos 3 (39.5), HunyuanVideo 1.5 (33.4), Wan 2.2 A14B (32.2).
• Internal benchmark: state-of-the-art among open-source competitors on TI2V (both general quality and embodied domain); ranks second in T2V general quality but still leads embodied-domain score there.
• Human GSB study (400 prompts/pair): Good-rate exceeds Bad-rate vs. most open-source baselines in T2V and vs. all open-source baselines in TI2V, though it trails top commercial models (Kling-V3, Wan 2.7, Seedance 2.0, HappyHorse 1.0).

Why it matters / caveats: It is presented as the first large-scale, open-source MoE video foundation model targeting the gap between video generation and embodied/robotic use, with released code and checkpoints; however, the 120B-parameter scaling runs were not trained to full convergence due to resource constraints, and the model still trails leading closed-source commercial systems in human evaluation.

Problem: Interactive world models (autoregressive video generators that react to user actions) typically degrade after seconds to minutes due to compounding rollout errors, and achieving high-fidelity real-time interactivity at that duration has remained computationally impractical.

Method: The authors present LingBot-World 2.0 / LingBot-World-Infinity, a causal (autoregressive) Diffusion Transformer video world model trained with a two-stage pipeline: (1) pretraining with a "Mixture of Bidirectional and Autoregressive Attention" (MoBA) mask that blends teacher-forcing causal self-attention with a bidirectional regularizing component, plus causal cross-attention over Plücker-embedding camera poses and chunk-wise text prompts, optimized via conditional flow-matching; (2) post-training distillation combining consistency distillation (reducing denoising steps) with distribution matching distillation (DMD) applied over long self-rollout trajectories to suppress drift. Deployment adds systems-level optimizations (compiler/kernel fusion, hybrid parallelism, async latent-to-frame streaming, dynamic KV-cache scheduling, a TensorRT spatio-temporal refiner) and an agentic "Director-Pilot" harness where a VLM (Director) proposes semantic events/object interactions (via SAM-based tracking for object-centric mode) and the diffusion generator (Pilot) renders them. Models are released at 14B and 1.3B parameter sizes; training data comes from a curated pipeline of egocentric video, synthetic game/Unreal Engine data, and web video with multi-track, chunk-wise VLM annotations.

Key results:

• Sustains real-time generation at 720p and 60 fps.
• Demonstrated a single uninterrupted one-hour (60-minute) generation session spanning 20 distinct scenarios with no perceptible visual quality decay.
• Claimed to be the only compared system achieving hour-level/"infinite" generation duration in a general domain, versus "minutes" for Matrix-Game 3.0, DreamX-World, LingBot-World 1.0, HappyOyster, and Genie 3.
• Qualitative comparisons show the model matching or exceeding visual quality of closed-source baselines (HappyOyster, Genie 3) and outperforming open-source causal backbones (WorldPlay, MAGI-1, HY-World 1.5, SANA-WM) in long-horizon stability, though no numeric benchmark scores (e.g., FID/VBench/user-study percentages) are reported in the text.

Why it matters / caveats: It is presented as an open, fully released alternative to closed systems like Genie 3 that pushes interactive world models from minute-scale to hour-scale stable generation while adding a rich action space (combat, archery, spell-casting, environmental events) and an agentic harness for goal-directed world evolution. The authors acknowledge key caveats: the model lacks true long-term memory (revisited regions are regenerated, not recalled), character identity/art style can drift over very long rollouts, physical understanding is imperfect (objects can interpenetrate), and real-time inference still requires substantial compute, not yet commodity hardware.

Problem: Existing robot-manipulation benchmarks are limited to simple/short-horizon/skill-narrow tasks and evaluate policies in either simulation or the real world alone, preventing systematic, reproducible, comprehensive assessment of generalist manipulation policies.

Method: RoboDojo pairs a simulation benchmark (42 tasks on the ARX X5 bimanual platform spanning five capability dimensions: Generalization, Memory, Long-Horizon, Precision, and Open) built on NVIDIA Isaac Sim/Isaac Lab with heterogeneous parallel simulation (differing scenes/objects run concurrently across vectorized environments) and a real-world benchmark of 18 tasks across three embodiments (ARX X5, Piper, Piper X) run through RoboDojo-RealEval, a standardized hardware/software rig (fixed camera/robot poses, touchscreen reset/scoring interface, layout-replay overlay, remote cloud evaluation). XPolicyLab supplies a shared observation-action policy interface so the same model can be trained/evaluated across both settings with minimal adaptation; simulation data (3,500 trajectories, 1.86M frames) is collected via cuRobo v2-based automated skill synthesis plus VR teleoperation, and real-world data (1,800 trajectories, 1.61M frames) via homogeneous leader-follower teleoperation. Leaderboard integrity is enforced via public + hidden verification layouts, multi-seed reporting, and released checkpoints/code/videos for verified submissions.

Key results:

• 30 policies integrated via XPolicyLab and evaluated on the simulation leaderboard (2,100 episodes/policy); best policy (Hy-Embodied-0.5-VLA) reaches only 8.80% average success rate / 13.07 average score vs. human expert teleoperation at 76.03% / 80.42.
• Per-dimension leaders: Spatial Forcing best on Generalization (9.33%/14.12), X-VLA best on Precision (12.00%/18.32), Hy-Embodied-0.5-VLA best on Long-Horizon (14.92%/25.74) and Memory (12.11%/13.37); Open dimension is near-unsolved, with the best policy (π0.5) at only 1.67%/1.98.
• Standard-vs-randomized scene comparison: performance collapses broadly under domain randomization (e.g., Hy-Embodied-0.5-VLA drops 92.9%, from 21.98 to 1.57); Spatial Forcing degrades least (67.2% drop), attributed to explicit 3D spatial grounding.
• Real-world leaderboard (10 policies, 180 trials each across 18 tasks/3 embodiments): best policy π0.5 achieves only 12.8% success rate / 22.9 score vs. 100%/100 for human teleoperation; sim and real-world rankings are only partially aligned (e.g., InternVLA-A1 and GalaxeaVLA rank higher in real-world than in sim).
• Heterogeneous parallel simulation gives a 1.94x throughput speedup over non-heterogeneous parallel sim under zero-action rollouts (77.4 vs. 40.0 interactions/s on 8×RTX 4090) and 1.63x with π0.5 inference included (64.0 vs. 39.2 interactions/s); full 18-task real-world evaluation completes in 202.0 minutes.

Why it matters / caveats: RoboDojo shows current generalist VLA/WAM policies remain far from human-level performance (often <15% success even on the "easier" dimensions) and exposes fragmented, dimension-specific progress plus real-world-specific failure modes (action jitter, unsafe behaviors) not visible from simulation alone; the authors explicitly caveat that sim and real-world tasks are not paired one-to-one, so the benchmark is not meant to measure direct sim-to-real transfer.

Problem: There is no real-world, city-scale, continuous multimodal dataset that lets researchers build and evaluate photorealistic urban digital twins and embodied spatial-reasoning agents at the scale of an entire city.

Method: The authors release WildCity, a dataset of 18 trajectories (avg. 83.7 km, 1,507.1 km total, 3.01M keyframes) collected by autonomous-fleet vehicles across six US cities using 6 surround-view RGB cameras, roof LiDAR, IMU, and GPS, with SLAM+GPS-refined poses, motion-compensated LiDAR, and SAM3-generated (validated against manual annotations) ground/sky/dynamic-object masks. On top of this they build an urban-tailored 3D Gaussian Splatting (3DGS) reconstruction baseline that jointly optimizes per-keyframe ego poses and rig camera extrinsics, models sky with a separate view-dependent MLP, applies ground-Gaussian regularization losses (height-variance, vertical-alignment, opacity), uses a progressive Difix3D+ render-repair-augment loop for extrapolated views, and shards Gaussians across GPUs (following GrendelGS/gsplat) to scale to billions of primitives; the reconstructed scenes are then plugged into a closed-loop simulator driven by a Vision-Language-Action model (Alpamayo) for embodied navigation.

Key results:

• Dataset stats: 18 logs across 6 cities (Atlanta, Arlington, Ann Arbor, Eden Prairie, Grand Rapids, Miami), ~40.18 km² coverage per city, ~2.5 hours per log, semantic masks reach 91.58% overall mIoU vs. 100 manually annotated images.
• On Ann Arbor-0.5k: their method reaches PSNR 29.99 / SSIM 0.917 / LPIPS 0.240 / Depth-L1 15.158 m, vs. H-3DGS (27.59/0.849/0.318/18.450) and CityGaussianV2 (24.41/0.856/0.409/17.053).
• On Atlanta-5k (2.5 km): their method gets PSNR 23.14 / D-L1 6.622 m, the best among baselines, vs. CityGaussianV2's D-L1 8.349 m and H-3DGS's 14.368 m.
• VGGT-Long+CityGS (feed-forward pose/point priors replacing SLAM) collapses to PSNR 13.40 dB / D-L1 20.934 m on the long trajectory, showing feed-forward 3D priors cannot yet substitute globally consistent sensor-based pose optimization at city scale.
• Ablations show removing sky modeling causes Depth-L1 to blow up to 148.54 m on Atlanta-0.25km, while removing ground regularization or pose optimization degrades geometry despite sometimes similar 2D metrics.
• Loop-closure analysis reports sub-centimeter horizontal drift and centimeter-level vertical drift per kilometer.

Why it matters / caveats: WildCity is presented as the first real-world dataset combining multi-city coverage, long continuous traversals, surround-view sensing, and city-scale route length, giving a more realistic (noisy, dynamic, imperfect-pose) benchmark than synthetic city datasets like MatrixCity; the authors note current reconstruction methods still degrade with scale, struggle under view extrapolation, and remain sensitive to real-world uncertainty (dynamic objects, lighting, pose noise), and that semantic masks and pose estimates, while validated, still carry residual errors.

Problem: Embodied agents (navigation, EQA, manipulation) are hand-designed compositions of perception, memory, planning, and action modules, and it is unclear whether Agent Architecture Search (AAS)—shown to work for text-domain LLM agents—transfers to perceptual agents evaluated via simulator rollouts.

Method: The authors introduce AgentCanvas, a typed-graph runtime that represents an embodied agent as an editable node-and-wire JSON graph with typed ports, pre-rollout type-checked patches, and episode-level logging (backed by Python node modules, elastic batched FM calls across simulator workers); and KDLoop, a coding-agent search procedure cycling through THINK (propose up to 3 axis-tagged experiments), CRITIC (check against past failures/constraints), EXPERIMENT (apply/evaluate), and DISTILL (write typed memory: confirmed findings, refuted edits, coverage), with a triggered REFLECT phase after stalls. ADAS and AFlow are ported into the same coding-agent harness (Claude Code as the editing session, Claude Opus 4.7 with 1M-token context as orchestrator) for direct comparison, so only proposer/memory logic differs across variants. Search is "method-seeded," starting from four published architectures (MapGPT, SmartWay for VLN; ExploreEQA for EQA; VoxPoser for manipulation) rather than building agents from scratch, with GPT-5-mini as backbone (except VoxPoser).

Key results:

• 3×4 optimizer×executor matrix (ADAS, AFlow, KDLoop × MapGPT, ExploreEQA, SmartWay, VoxPoser), success rate (SR) reported as mean±sd over 3 post-selection reruns.
• MapGPT: baseline 46.9±3.1% SR; AFlow reaches 54.5±3.1% (+7.6pp), KDLoop reaches 54.0±2.3% (+7.1pp), ADAS reaches 49.1±3.2% (+2.2pp).
• ExploreEQA: baseline 43.0±1.7%; AFlow +4.7pp (47.7±2.1%), KDLoop +3.0pp (46.0±1.0%); ADAS found no improving graph-level edit.
• SmartWay: baseline 29.7±2.1%; ADAS +4.0pp (33.7±2.5%); AFlow's apparent +9.0pp (38.7±5.9%) is flagged as leak-bearing (it wired a ground-truth Habitat evaluator signal into the graph) and excluded from deployable-gain claims; KDLoop +1.3pp (31.0±4.6%).
• VoxPoser: baseline 9.0±0.0%; ADAS +3.7pp (12.7±0.0%), AFlow +3.9pp (12.9±0.5%); KDLoop made no mutation, instead detecting a deliberately injected substrate-level logging fault (missing LMP sub-call self-reports) — the only variant of the three to do so.
• ADAS/AFlow concentrate ~60%/43% of edits on a single intervention axis and run to the iteration cap in all 8 cells (84 and 82 total iterations), evidencing "local edit basin" convergence; ADAS rediscovers the identical MapGPT score (SR=0.4769) across six iterations.

Why it matters / caveats: The paper shows AAS can yield real, confirmed SR gains on embodied agents (not just superficial workflow restructuring), but exposes embodied-specific failure modes largely absent in text-domain AAS: rollout noise can make single-pass "best" scores unreliable, search gets stuck in local edit basins without axis-level memory, and episode-level credit assignment (diagnosing why a score changed, e.g., detecting evaluator leakage or logging faults) only partially emerges even with full logs — KDLoop's typed memory helps but is not a complete fix.

Problem: Large Language Models perform poorly at code completion for severely low-resource programming languages like Pharo (only ~2k public repositories), leaving its IDE limited to single-token completion.

Method: The authors build an end-to-end pipeline that curates Pharo training data from 415 MIT-licensed, Tonel-format GitHub repositories (387,159 extracted methods, using a custom Pygments lexer and tree-sitter grammar), then apply continued pre-training (causal language modeling plus AST-aware fill-in-the-middle masking) followed by LoRA-based supervised fine-tuning (Random-AST masking) on Qwen2.5 Coder Base (0.5B/1.5B/3B/7B) and Mellum-base (4B). They evaluate with new benchmarks: Pharo translations of HumanEval+ and Exercism (method-level, AST-aware and Random-AST variants) and a repository-level benchmark of 2,185 masked-span completion tasks from 488 real GitHub commits, tested under four context strategies (no context, class signatures, package signatures, impacted methods), comparing pass@1, ChrF, and CrystalBLEU against base checkpoints and against Qwen3 Coder 480B A35B Instruct and Claude Sonnet 4.5.

Key results:

• Qwen2.5 Coder 3B - SFT improved HumanEval+ AST-aware pass@1 from 71.48% (base) to 83.73%; Qwen2.5 Coder 7B - SFT reached 89.04% (from 71.12%, +17.92 points).
• On the more realistic Random-AST (r-AST) HumanEval+ setting, Qwen2.5 Coder 7B - SFT rose from 45.24% to 52.76% pass@1, exceeding both Qwen3 Coder 480B A35B Instruct (45.13%) and Claude Sonnet 4.5 (51.53%) — a >60x-larger model advantage overturned.
• Base LLM failures were mostly syntax errors (65.6%), unexpected exceptions (17.9%), assertion failures (16.5%); the two-step training cut syntax errors by ~33% on average.
• Repository-level: providing "impacted methods" (other methods changed in the same commit) as context was the most effective strategy, boosting Qwen2.5 Coder 7B - SFT ChrF from 60.05% to 75.96% (+15.91) and CrystalBLEU from 35.96% to 58.99% (+23.03); even the 1.5B specialized model with this context beat Qwen3 Coder 480B A35B Instruct, and the 3B/7B models beat it by +7.52% ChrF / +8.63% CrystalBLEU on average.
• Claude Sonnet 4.5 remained the top model overall (ChrF 83.02%, CrystalBLEU 70.52% with impacted-methods context) but showed suspiciously high similarity even with no context, suggesting possible data contamination.
• 4-bit (Q4_K_M) quantization cut the 7B model's memory footprint ~70% (14.19 GiB to 4.36 GiB) with only a 0.61% average pass@1 drop; inference latency was ~0.5-0.7s for the 3B model and ~1.3s for the quantized 7B model on consumer hardware (Apple M3/M4 Max, RX 7800XT).

Why it matters / caveats: The study demonstrates that domain specialization of small (0.5B-7B) open code LLMs can match or beat general-purpose models orders of magnitude larger for a severely low-resource language, enabling real-time, locally deployable in-IDE completion; caveats include unavoidable residual risk of data contamination in baselines, repository-level evaluation relying on similarity metrics (ChrF/CrystalBLEU) rather than executable tests, and no live user study in the actual Pharo IDE yet.

Problem: Existing asynchronous RL pipelines for LLMs improve GPU efficiency over synchronous training but leave stability and effectiveness underexplored, since group-wise sampling methods like GRPO are structurally mismatched to asynchronous training (they must wait for the slowest rollout in a group and are incompatible with single-trajectory-feedback agentic/online settings).

Method: SAO (Single-rollout Asynchronous Optimization) replaces GRPO's group-wise sampling with single-rollout-per-prompt updates and stabilizes training via Direct double-sided Importance Sampling (DIS): it computes the probability ratio directly from rollout-engine log-probabilities (dropping the stale "old policy" model) and applies strict double-sided token-level clipping/masking (interval [1-ϵl, 1+ϵh]) that zeroes out gradients for tokens outside the trust region. To make single-rollout practical, it uses a value-model recipe with "Faster Value Update" (K=2 critic updates per policy update), "Frozen-Attention" value-model training (freezing attention, updating only MoE projections), scaled value-model pretraining, and a "Skip-Observation" token-level GAE that bridges advantage computation across action boundaries while skipping environment-feedback (observation) tokens. Experiments use Qwen3-30B-A3B-Thinking-2507 (fine-tuned on GPT-OSS-120B-generated TIR data) as backbone; SAO was also deployed in training the open GLM-5.2 model (750B-A40B).

Key results:

• SAO reaches 97.3% (AIME2025), 74.8% (BeyondAIME), 88.3% (HMMT Nov 2025), 74.0% (IMOAnswerBench) vs. GRPO's 84.2%/54.8%/76.0%/55.8% respectively, and vs. SFT baseline 80.4%/53.3%/75.2%/53.3%.
• On SWE-Bench Verified: SAO 29.8% vs. GRPO(w/ DIS) 27.0% vs. Qwen3-30B-A3B base 23.0%.
• Vanilla GRPO collapses at ~160 training steps; GRPO with DIS is stabilized but SAO diverges further ahead after ~400 steps; SAO trains stably for ~1000 steps.
• Ablations: removing faster value update drops AIME2025/BeyondAIME to 95.0/69.8; removing frozen-attention drops to 90.6/74.5; vanilla VAPO (no DIS) collapses around step 90 and scores only 91.3/69.0; a running-mean-baseline (no value model) scores just 79.8/55.3.
• Step-level GAE variants underperform token-level GAE (89.8/66.8 at 400 steps) — Step-Average 85.8/60.5, Step-Last-Token 87.3/62.8.
• In a simulated online learning task (shifting stylistic reward preferences), SAO's value-based critic adapts rapidly to reward shifts, recovering faster and to higher reward levels than a running-mean-baseline approach.

Why it matters / caveats: SAO shows single-rollout, value-model-based RL can outperform group-based methods like GRPO in asynchronous, agentic, and non-stationary/online settings where per-prompt single-trajectory feedback is the norm, and it was validated at scale in GLM-5.2 training. Caveats: experiments center on a Qwen3-30B-A3B backbone for agentic reasoning/coding/simulated writing tasks, so conclusions may not transfer to smaller models or dense-reward/short-rollout environments; SAO depends on a well-trained value model and reliable rollout log-probabilities.

Problem: Linear RNNs (e.g., Gated DeltaNet, Mamba2) have fixed, small hidden states that are cheap but limit long-context recall relative to transformers, and simply enlarging the dense state makes per-token FLOPs grow linearly, making it impractical.

Method: Sparse Delta Memory (SDM) extends Gated DeltaNet by replacing the dense key-value outer-product state update with a large explicit memory table (N slots) addressed via a Product-Key-Memory-style sparse indexing scheme, performing gated delta writes to only W selected slots and reads from R selected slots (with W=R=64, matching GDN's FLOPs/parameters under an isoFLOP design). The initial memory state M0 is treated as a learned parameter (rather than null-initialized), letting the model store pretraining knowledge in the large state at no extra inference cost. Models use a hybrid architecture (3:1 sliding-window-attention to global-layer ratio, GQA, RoPE, gated MLP) trained with WSD schedule/AdamW, evaluated across a scaling ladder up to 8B activated parameters on >1 trillion tokens, with long-context fine-tuning on 128k-token sequences.

Key results:

• SDM outperforms iso-FLOP GDN training loss at every scale on the ladder (R²=0.999 scaling-law fit); at 8B scale SDM reaches lower loss than GDN and even outperforms a FullAttn 8B model.
• Short-context: at 1.4B, SDM improves over GDN on 13/15 reasoning/knowledge tasks (avg accuracy 38.51 vs 37.93 GDN, vs 37.98 FullAttn); at 8B, SDM average accuracy 56.84 vs GDN 55.70 and FullAttn 56.65; SDM achieves lowest DCLM/validation NLL among all models at both scales.
• RULER long-context recall: SDM average accuracy 31.2 (1.4B) and 50.2 (8B) vs GDN 20.0/34.2 — improves on 6/6 evaluated RULER task categories; FullAttn reaches 32.5/61.2 thanks to unbounded KV cache, but SDM matches/exceeds FullAttn on 4/6 tasks at 1.4B and 3/6 at 8B.
• Code-data perplexity: SDM perplexity drops to ~2.0 at long context (32k-1M tokens) vs ~2.2-2.3 for Mamba2/GDN.
• Ablations: removing learned M0 still leaves SDM far above GDN (confirms gains mainly from state size, not initialization); NLL degrades monotonically as memory shrinks from 432MB to 27MB (0.914→0.947), though even the 27MB SDM beats GDN on RULER.
• Efficiency: SDM state is ~4,000x larger than GDN's at 8B scale, yet end-to-end training is only 1.49x slower than GDN; 1.4B inference decode is ~10% slower than GDN but 6x faster than FullAttn.

Why it matters / caveats: SDM shows that sparsity lets linear RNNs scale state size by orders of magnitude at constant FLOPs, closing much of the long-context recall gap with transformers while retaining constant per-token compute/memory — but the memory footprint can rival total model parameters, current SDM kernels have much lower MFU than optimized GDN kernels, and it still trails FullAttn substantially on some RULER subtasks.

Problem: Long-horizon failure in world models is conventionally attributed to generic "compounding error," but this framing fails to distinguish whether imagined rollouts fail because errors compound or because the model imagines kinematically (linear position-velocity-acceleration extrapolation) rather than dynamically (respecting mass, friction, contact, and other physical constraints).

Method: The authors define imagined Kinematic-Consistency Error (iKCE), a per-step diagnostic measuring how far each predicted next state in an imagined rollout departs from a closed-form kinematic predictor (e.g., constant-velocity extrapolation) applied to the previous state, adapting the KCE loss from Gao et al.'s StyleVLA training objective into a test-time diagnostic. This is paired with a conditioning-perturbation protocol that sweeps physically meaningful parameters (e.g., surface friction) across a regime boundary (defined empirically as where policy reward drops below 50% of baseline) and checks whether iKCE responds (dynamic signature) or stays flat (kinematic signature). The diagnostic is instantiated on a released open-weight DreamerV3 checkpoint trained on DMC walker-walk, comparing imagined rollouts against matched real-physics rollouts under the same trained actor.

Key results:

• iKCE of imagined DreamerV3 rollouts is ~180x higher than matched real-physics rollouts at horizon T=16 (7.7×10⁻³ vs 4.2×10⁻⁵), and ~30x higher at T=64 (2.6×10⁻³ vs 8.6×10⁻⁵).
• Across a 13-point friction sweep crossing the empirical gait-collapse boundary at µ=0.20, WM-imagined iKCE stays statistically flat (log-log slope βWM=−0.009, CI [−0.096, +0.082], contains zero) even as the trained policy's episodic reward collapses from ~650 (µ=0.5) to ~200 (µ=0.10).
• Real-physics iKCE under the same sweep shows a genuine friction response (βphys=−0.220, CI [−0.301, −0.142], excludes zero).
• Findings are robust across 3 additional seeds, an actor-retrained-at-horizon-64 control, a domain-randomization control (βWM_DR=−0.026 vs βphys_DR=−0.114), an alternate kinematic slice, and a joint-noise positive control.
• A horizon-emergence test shows the dynamic (friction-sensitive) signature in physics only appears at longer horizons (T=32–64), while the WM slope stays near zero at every horizon.
• A long-horizon-trained actor (imagination horizon=64) converges to only ~400 episodic reward vs ~955 for the default h=15 checkpoint, with training instability.

Why it matters / caveats: The results suggest that policy gradients propagated through long imagined DreamerV3 rollouts optimize against a trajectory distribution that diverges from real dynamics in ways iKCE-relevant regime changes expose, potentially explaining unstable long-horizon actor training. Caveats: the result rests on a single embodiment (2D 9-DOF DMC walker-walk) and a single world-model family; the physics-side policy-out-of-distribution confound is only partially bounded. Code, checkpoints, and sweep data are released (github.com/TUM-AVS/iKCE). "