AI papers — 2026-07-01

Problem: Existing models are optimized for isolated next-token, next-frame, or next-action prediction, but there is no unified world foundation model that learns a general latent world-state representation supporting understanding, prediction, and action simultaneously.

Method: Orca is an Encoder-Decoder world foundation model built on a native pre-trained VLM (Qwen Team, 2026a) backbone that learns a unified world latent space via Next-State-Prediction, combining "unconscious learning" (observation-only state transition from dense video frames via teacher-forced MLP prediction against a frozen vision encoder's latents) and "conscious learning" (event-conditioned state transition using language-described instructions, plus VQA response generation via the LM head). After pre-training, the Orca backbone is frozen and only lightweight modality-specific readout modules are trained: a reused LM head for text, an MLP adaptor + LoRA on frozen Stable Diffusion 3.5 (MMDiT) for image prediction, and an MLP adaptor + DiT-based flow-matching Action Expert (trained from scratch) for embodied action generation.

Key results:

• Pre-training inventory: 125K hours of video data, 160M event annotations, and 11.5M general VQA samples (only one-tenth of video data used in this version).
• Text generation (avg of MVBench, TemporalBench, 3DSRBench, SWITCH): Orca-4B scores 51.8 vs. Qwen3.5-4B at 46.7, Gemma 4-4B at 40.8, and world models Emu3 (30.4) and Emu3.5 (29.8).
• Cross-benchmark capability gains for Orca-4B vs Qwen3.5-4B: State Transition +12.27%, Commonsense Reasoning +5.19%, Spatial Relations +0.57%, Dynamic Motion +8.52%.
• Image prediction (PRICE-V0.1 benchmark, avg across 4 judge models): Orca (4B+2B) achieves 59.8±10.9, best among baselines including FLUX.2 [klein] (56.1±18.1), FLUX.1-Kontext (40.9±13.5), and OmniGen2 (39.6±10.2).
• Action generation (real dual-arm robot, 5 OOD tasks): Orca achieves overall Rule-based score of 32.4 vs. π0.5 at 29.4, Qwen3.5 w/ Action Expert at 10.5, and V-JEPA 2.1 w/ Action Expert at 17.0, going from a 0% success rate baseline (Qwen3.5) to comparable/competitive performance with the pre-trained π0.5.
• Infrastructure (FlagScale with FSDP2, chunked cross-entropy loss, communication pre-fetching): training throughput improved from 0.66 to 2.91 Samples/Sec/GPU, a ~4.4x acceleration versus StarVLA.
• Ablation (Table 5): using all three losses (λobs, λevt, λvqa) together yields the best balanced average score of 48.0, versus 29.3–44.6 for partial-loss combinations.

Why it matters / caveats: The authors frame Orca as an early exploratory milestone toward general-purpose world foundation models, showing that a single frozen world latent can support text, image, and action readouts with performance scaling with data/model size. Stated limitations include: signals limited to vision and language only (missing audio, tactile, force, etc.), state supervision tied to a frozen ViT's semantic space rather than a truly native world space, model scale capped at 4B/0.8B with only one-tenth of collected data used, a limited-scope PRICE-V0.1 benchmark, short-horizon (minute-level) event annotations insufficient for long-term state evolution, and embodied tasks that remain relatively short and easy despite stringent evaluation settings.

Problem: Execution-based verification of coding-agent patches requires costly per-repository Docker environments (dependency setup, test discovery, test runners), which is infeasible for many real-world repositories that lack reproducible environments or test suites, and existing environment-free verifiers only score patches using surface-level information without inspecting the repository.

Method: Dockerless is an agentic, environment-free patch verifier that, given an issue, reference patch, and candidate patch, first generates K (2-4) verification questions, dispatches parallel sub-agents that explore the codebase with read-only shell tools (find, grep, rg) to produce evidence-backed answers, and then aggregates the question-answer evidence to output a binary verdict token whose logits define a continuous correctness score. It is trained via rejection sampling on 3.7K execution-labeled issues from SWE-Gym and Multi-SWE-RL (keeping only question-answer-judge trajectories whose predicted verdict matches ground truth), using Qwen3.5-9B as the backbone, and is then used as both the SFT trajectory filter (environment-free RFT) and the RL reward source (via GRPO) for a fully environment-free post-training pipeline.

Key results:

• Dockerless reaches 81.0 AUC on SWE-bench Verified and 72.1 AUC on Multi-SWE-bench Flash on the verifier evaluation benchmark, beating the strongest open-source trained verifier (DeepSWE Verifier) by 14.3 and 9.2 AUC points, and the strongest frontier LLM judge (GPT-5.4/GLM-5) by 5.1 and 8.2 points respectively.
• The fully environment-free post-training pipeline (Dockerless-RL-9B) reaches 62.0%, 50.0%, and 35.2% resolve rate on SWE-bench Verified, Multilingual, and Pro, improving over the Qwen3.5-9B baseline by +2.4, +8.7, and +2.9 points, and over the next-best open-source specialist (SWE-Lego-8B) by +20.8, +31.0, +19.1 points.
• Dockerless-filtered SFT (top 4K of 16K env-free trajectories) nearly matches environment-based SFT (60.6 vs 60.0 on Verified, 47.7 vs 48.3 on Multilingual, 35.3 vs 33.9 on Pro), while training on all 16K unfiltered trajectories underperforms the base model (58.8/41.3/31.9).
• Env-free RL with Dockerless rewards nearly matches oracle test-execution RL (62.0 vs 62.4 Verified, 50.0 vs 51.3 Multilingual, 35.2 vs 35.7 Pro) and outperforms RL using DeepSWE Verifier rewards by +1.4/+2.7/+1.1 points.
• Verifier AUC rises from 78.3 (K=0 questions) to a peak of 81.0 (K=4), then fluctuates (79.6 at K=6, 80.3 at K=8), motivating the chosen K=2-4 setting.
• In RL training latency analysis (7680 rollouts), agent rollouts average 2308s versus only 41-180s for reward evaluation; Dockerless reward computation adds 180s (7.2% of total per-rollout time).

Why it matters / caveats: By matching environment-based post-training performance while requiring zero per-repository setup, Dockerless demonstrates a scalable path to post-train coding agents on the long tail of real-world (private, enterprise, legacy) repositories that lack reproducible test environments; the paper also notes that removing the environment during rollout collection alone (without a strong verifier) costs frontier models only 3.0-13.9 resolve-rate points, indicating the verifier—not the agent rollout—was the key bottleneck being addressed.

Problem: On-policy distillation methods that inject privileged information into teacher or student policies to raise the distillation ceiling can suffer from "privilege illusion," where apparent teacher-student performance gains reflect unlearnable information asymmetry rather than transferable capability, and this is worsened because most tokens carry little capability-bearing signal while a small subset carries the pivotal ones, yet vanilla on-policy distillation (OPD) supervises all tokens uniformly.

Method: DOPD (Dual On-policy Distillation) is an advantage-aware dual distillation paradigm that computes a "privilege advantage gap" A (the absolute log-probability difference between a privileged teacher policy and a privileged student policy on each token) and combines it with each policy's predicted token probability to sort every token into one of four regimes: low-gap/high-probability (light Top-K reverse-KL teacher distillation), low-gap/low-probability (weak Top-K reverse-KL self-regularization to the privileged student with stop-gradient), high-gap/high-teacher-probability (full-vocabulary JS-divergence teacher distillation), and high-gap/high-student-probability (light Top-K reverse-KL to the privileged student). Experiments use Qwen3 and Qwen3-VL model families (e.g., Qwen3-8B to Qwen3-1.7B for LLM, Qwen3-VL-8B to Qwen3-VL-2B for VLM), with privileged information generated by GPT-5.4 (step-wise decomposition hints for LLM tasks, bounding boxes with object labels for VLM tasks), trained on a mixture of RaR-Science-20K, DAPO-Math-17K, Skywork-OR1-Coding-14K (LLM, 32K samples) and ViRL39K (VLM, 25K samples).

Key results:

• On 8 LLM benchmarks (C-Eval, LiveBench, MATH500, AIME25, ZebraLogic, AutoLogi, BFCLv3, LCBv5), DOPD reaches 51.4 average vs. 43.9 for Vanilla OPD, a 7.5-point gain, closing 89.8% of the teacher-student gap (12.3-point absolute recovery), and outperforms the strongest baselines ExOPD/Uni-OPD/EOPD by 4.4/4.8/5.3 points on average.
• On 8 VLM benchmarks (RealWorldQA, MMStar, MathVision, DynaMath, LogicVista, MMMU, MMMU-Pro, VSI-Bench), DOPD reaches 58.4 average vs. 52.4 for Vanilla OPD (a 6.0-point gain, 10.1-point absolute gain over student, 69.2% gap recovery), beating Uni-OPD/Vision-OPD/VA-OPD by 4.2/2.8/2.1 points.
• Across five teacher-student size pairs (Qwen3-8B/4B/1.7B → Qwen3-0.6B, Qwen3-8B/4B → Qwen3-1.7B), DOPD improves 6.2-10.6 points over Vanilla OPD and gains 11.1-14.1 points on average, roughly 2-3x Vanilla OPD's improvement; in the largest mismatch (Qwen3-8B→Qwen3-0.6B), Vanilla OPD gains only 3.5 points versus DOPD's 14.1-point gain (53.0% gap recovery).
• On out-of-distribution evaluation, DOPD outperforms the second-best baseline by 3.1 and 4.3 points (reasoning and coding transfer respectively).
• Ablations: using only high-teacher/low-student-probability tokens already beats equal-weight distillation of all four token types (Vanilla OPD equivalent) by 4.6 points; full adaptive routing over all four token types yields a further 8+ point improvement. Privileged information choice matters: step-wise hints without execution traces yield the largest LLM gains (8.3 and 10.4 points on C-Eval/LiveBench over the no-privilege baseline), while directly giving the final answer as privileged input underperforms even the no-privilege baseline (59.5 C-Eval).
• Best distillation intensity coefficients found via sensitivity study: weak βw = 0.3, light βl = 0.6; Top-K set to K=128.

Why it matters / caveats: DOPD provides consistent, scalable gains (especially for large teacher-student capacity mismatches) plus better training stability, continual learning, and OOD generalization than vanilla or single-sided (self/adaptive) OPD variants. The authors note limitations: it depends on the availability and quality of privileged information (added annotation/generation/filtering cost), requires an extra forward pass of the student model versus Vanilla OPD, and its token-routing mechanism, while empirically stable, is still heuristic rather than learned.

Problem: Existing diffusion-based speculative decoding methods (e.g., DFlash) use a fixed inference block size inherited from training, even though the optimal block size actually varies from sample to sample.

Method: The authors analyze block-diffusion speculative decoding (where a diffusion draft model, dLLM, proposes B tokens per step that are verified by an autoregressive target LLM) and find that the per-sample optimal block size B*(x) clusters locally around the training block size B (within a range of about ±3). Based on this, they propose BlockPilot, which formulates block-size selection as a classification problem: after the target model's prefilling stage, the predictive probability distribution of the last token is fed into a lightweight two-layer MLP ("block size predictor," hidden dim 2048) that outputs a softmax over a local candidate set {B−k,...,B+k} (default k=2), trained with cross-entropy loss on supervised data constructed via offline enumeration of acceptance length τ(b;x). The predicted block size is fixed once (after prefilling) and used for the rest of the diffusion draft/verify cycle, requiring no changes to the draft or target models.

Key results:

• On Qwen3-4B at temperature T=1, BlockPilot achieves an acceptance length τ of 5.92 and a 4.20× speedup, versus DFlash(16)'s 3.80× and DFlash(32)'s 2.61×.
• On Qwen3-8B at T=1, BlockPilot reaches 3.94× speedup (τ=5.55), versus best fixed-block DFlash(16) at 3.55×.
• At T=0, BlockPilot improves average speedup from 3.99× (DFlash(16)) to 4.17× on Qwen3-4B (τ from 6.31 to 6.59), and from 4.42× to 4.66× on Qwen3-8B (τ from 6.13 to 6.46).
• The predictor itself adds only 0.32B params, 0.02MB/0.62GB memory, and 7.34ms latency, versus 183–278ms latency for the backbone target models (Qwen3-4B, Qwen3-8B, Llama-3.1-8B-Instruct).
• Ablations show k=2 (candidate interval radius) and a 2-layer, hidden-dim-2048 MLP predictor using the raw (non-normalized, non-softmax) last-token predictive distribution as input give the best speedup/τ trade-off, e.g. 4.76× speedup / τ=6.90 on GSM8K vs. 4.55× / 6.59 with softmax preprocessing.
• Using only Top-k probabilities as predictor input caused severe overfitting (training accuracy ~80% vs. test accuracy ~10%), motivating use of the full predictive distribution instead.

Why it matters / caveats: BlockPilot shows that the decoding policy (not just the draft model architecture) is a learnable, impactful lever for speculative decoding efficiency, and it is plug-and-play — integrating into existing diffusion-based speculative decoding frameworks (evaluated against EAGLE-3 and DFlash) with only millisecond-level overhead and no change to output distribution/quality.

Problem: Feed-forward 3D Gaussian reconstruction methods from unposed multi-view images produce dense, unstructured points/Gaussians with no native notion of object instances, forcing any object-level querying, editing, or reasoning to rely on post-hoc grouping or aggregation.

Method: The paper introduces instance-structured 3D token groups: a frozen geometry foundation model (VGGT) extracts multi-view features and pointmaps fused into context tokens; an image-anchor transformer (Danchor) cross-attends to these to produce anchor tokens (decoded into 3D Gaussians via a 2-layer MLP), and an anchor-grouping transformer (Dgroup) cross-attends to the anchor tokens to produce group tokens that compete for anchor ownership via softmax assignment. Training uses joint 2D supervision only (no 3D annotations): RGB rendering loss (MSE + LPIPS) shapes anchor tokens, while a Hungarian-matched Dice+BCE instance-mask segmentation loss (following DETR/Mask2Former-style matching) shapes group tokens, with a linear warm-up on the segmentation loss weight. For semantics, 2D foundation model (LSeg) features are distilled into a decomposed representation: a shared 512-dim group-level embedding plus an 8-dim anchor-level residual per Gaussian.

Key results:

• On ScanNet feature lifting (2 context views), the model achieves the best mIoU on source (0.661) and target views (0.657), versus Uni3R (0.540/0.558) and C3G (0.542/0.513), while cutting semantic storage from 8.4M scalars (Uni3R) to 59.4K.
• Class-agnostic instance segmentation (8 context views) achieves the best AP across the board: AP 0.235, AP50 0.438, AP25 0.564, beating per-scene optimized Gaussian Grouping (AP 0.139) and ObjectGS (AP 0.178), and feed-forward+optimization IGGT+LUDVIG (AP 0.122).
• Ablation on joint training: removing joint training (sequential Danchor→Dgroup) drops segmentation AP from 0.193 to 0.032; removing the λseg warm-up drops it further to 0.081.
• Zero-shot transfer to MipNeRF360 (ScanNet-trained, no fine-tuning): outperforms Uni3R on PSNR (16.52 vs 14.58), SSIM (0.408 vs 0.317), and LPIPS (0.439 vs 0.472).
• On RealEstate10K trained with SAM2 pseudo-labels (no human annotations), the model outperforms C3G on PSNR (22.85 vs 22.39), SSIM (0.746 vs 0.713), LPIPS (0.230 vs 0.259).
• Training cost: fine-tuning VGGT takes <6 hours on 4 H200 GPUs; the 2-view tokenizer trains ~20 hours on 4 RTX A6000 GPUs, the 8-view setup ~12 hours on 4 H200 GPUs; feature-lifting training takes <3 hours on 4 RTX A6000 GPUs.

Why it matters / caveats: By making object instances (fewer than 100 groups) rather than tens of thousands of Gaussians the native representational unit, the framework enables instance-level manipulation (removal, translation, insertion, transformation) and open-vocabulary 3D retrieval whose complexity scales with instance count rather than primitive count, without post-hoc masks or per-scene optimization. Stated limitations: reconstruction PSNR/SSIM trails per-pixel Gaussian baselines like Uni3R/LSM (a gap attributed to the compact token bottleneck by design); evaluation focuses on bounded indoor scenes, with the fixed cap of L=100 groups and model training likely needing revisiting for outdoor/large-scale scenes; a single shared group-level token may lack expressivity for complex/highly varied instances; and the framework assumes static scenes, with manipulation robustness under heavy occlusion or object contact left as future work.

Problem: In two-stage visual generative pipelines, the VQ tokenizer is trained and frozen purely for reconstruction before the autoregressive (AR) generator is trained on its discrete indices, leaving the tokenizer oblivious to which token distributions the AR model can actually learn to predict.

Method: GEAR (Guided End-to-end AutoRegression) jointly trains a VQ tokenizer and an AR generator end-to-end by reading out each position's codebook assignment two ways: a non-differentiable hard, one-hot branch that trains the AR with next-token prediction (NTP) and a REPA-style alignment loss, and a differentiable, temperature-scaled soft branch that carries a representation-alignment loss (aligned to a frozen encoder like DINOv2) back to update only the tokenizer. This avoids the instability/codebook collapse of a straight-through estimator (STE), since the NTP loss never touches the tokenizer; the method is demonstrated on LlamaGen/LlamaGen-REPA backbones and generalizes across VQVAE, LFQ, and IBQ quantizers as well as to text-to-image generation (Qwen3-1.7B-conditioned, GPIC corpus).

Key results:

• Up to 10x faster ImageNet gFID convergence versus the LlamaGen-REPA baseline (naive STE end-to-end training diverges, gFID≈105).
• On ImageNet-256 with CFG at 300 epochs, gFID improves from 6.00→4.95 (111M/Base), 3.15→2.95 (343M/Large), and 2.68→2.52 (775M/XLarge) versus LlamaGen-REPA.
• On GPIC text-to-image (390k steps, w/ CFG), FDD improves from 127.9→115.3 versus LlamaGen-REPA; at 50k steps FDD improves 279.6→256.9.
• Ablations: STE variant collapses to gFID 104.9/rFID 59.7; GEAR improves all tested quantizers (VQVAE gFID 14.72→10.63, LFQ 18.68→14.78, IBQ 20.25→12.97); best guidance temperature τ=0.1, alignment coefficient λ=0.5, alignment at AR layer 8; best CFG scale 1.5 yields gFID 3.388 (from 10.63 without guidance).
• Tokenizer's DINOv2-alignment (CKA) drops from 0.1727→0.1070 at the patch level (becomes less semantic) while reconstruction is preserved (rFID largely unchanged), and codebook usage becomes lower-entropy/more concentrated, whereas the AR's hidden states become more DINOv2-like per patch.

Why it matters / caveats: GEAR shows that shifting representation alignment from the tokenizer to the AR (rather than diffusion-style latent semanticization) yields faster, higher-quality discrete AR image generation while remaining a drop-in tokenizer improvement transferable to frozen-tokenizer pipelines. The paper notes GEAR still trails the best end-to-end diffusion model (REPA-E, rFID 0.28, gFID 1.12) because its 16x-downsampling discrete tokenizer's reconstruction ceiling (rFID 1.64) upper-bounds generation quality (gFID 2.52 w/ CFG), and closing this reconstruction gap is cited as the biggest lever for further improvement.

Problem: Existing Block Diffusion Language Models (BD-LMs) are trained under Teacher Forcing (single noisy block) or Discrete Diffusion Forcing (D2F, monotonic noisy suffix), and neither training regime matches the bounded, heterogeneous-noise running-set states actually encountered during Multi-Block Diffusion (MultiBD) inference, causing a train–inference mismatch that limits reliable inter-block parallel decoding.

Method: The paper formulates Multi-Block Diffusion Language Models (MBD-LMs) as a unified view generalizing BD-LMs via a bounded "running-set" of consecutive blocks decoded concurrently, with Teacher Forcing (TF) and Discrete Diffusion Forcing (D2F) as extreme cases. It introduces Multi-block Teacher Forcing (MultiTF), a post-training method that builds systematic and random noise-group layouts (up to max group size Gmax), applies a randomized "chain-uniform" block-level noise-scheduler within each group, and uses a Group-Aware Dual-Stream Mask to let noisy blocks in a group attend to each other while conditioning on a clean prefix. For inference, it proposes an optimized MultiBD engine based on a Block Buffer mechanism (with dummy/active/to-cache/in-cache slot states) that keeps input shapes static for CUDA Graph capture/replay while preserving prefix-cache reuse. Experiments apply MultiTF to LLaDA2-Mini, LLaDA2-Mini-DMax, LLaDA2-Mini-CAP, LLaDA2.1-Mini, and SDAR-8B-Chat backbones, including combination with the DMax token-to-token acceleration method.

Key results:

• MBD-LLaDA2-Mini increases average Tokens Per Forward pass (TPF) from 3.47 to 6.19 (+78.4%) and improves average accuracy from 79.95% to 81.03% across GSM8K, MATH500, MBPP+, and HumanEval+.
• Combined with DMax, MBD-LLaDA2-Mini-DMax reaches average TPF of 9.34 (+47.1% over LLaDA2-Mini-DMax under SingleBD) with only a 1.02 percentage-point accuracy drop (78.57% vs. 79.59%).
• Using the optimized inference engine, MBD-LLaDA2-Mini-DMax achieves 951.41 TPS on average vs. 781.50 TPS for LLaDA2-Mini-DMax.
• On SDAR-8B-Chat-b32, MBD-SDAR-8B-Chat-b32 raises average TPF from 2.54 to 4.46 (+75.6%) and average accuracy from 69.00% to 69.74%.
• Ablations (LLaDA2-Mini-DMax, averaged over HumanEval+/GSM8K) show full MultiTF (systematic + random layouts, chain-uniform scheduler) reaches accuracy 84.59%, TPF 9.87, AUP 805.34, versus a D2F-style monotonic scheduler which raises TPF to 8.76 but drops accuracy to 79.34%.
• Measured TPS gains (e.g., 517.16 to 745.92, a 1.44x improvement, for one setting; 779.49 to 926.67, a 1.19x improvement, for LLaDA2-Mini-DMax) are shown to track the theoretical TPF-gain predictions once per-forward-pass compute cost is accounted for.

Why it matters / caveats: MultiTF lets existing BD-LMs be post-trained into MBD-LMs that convert increased decoding parallelism (TPF) into real wall-clock throughput gains without retraining from scratch, and the approach is shown to generalize across multiple BD-LM backbones (LLaDA2.x, SDAR) and to stack with T2T-based acceleration (DMax); the paper notes that naive/training-free MultiBD alone already boosts TPF but can degrade accuracy, and that DMax-combined MultiBD still incurs a small (~1 percentage-point) accuracy cost.

Problem: Existing LLM-driven evolutionary search methods (test-time search and test-time learning) leave the discovery capability—knowing what to mutate, what to keep, and when to backtrack—external to the model itself, so this capability is discarded after each task and cannot transfer across new optimization problems.

Method: The paper introduces Evolution Fine-Tuning (EFT), a mid-training paradigm that converts evolutionary search trajectories into supervised training examples so an LLM learns to act as a mutation operator. The authors build the Finch Collection, a 156K-trajectory dataset (from 172,997 raw trajectories after filtering) spanning 10 domains and 371 optimization tasks, collected using the OpenEvolve search scaffold with Qwen3.5-397B-A17B as the teacher mutation operator; they then fine-tune Qwen3.5 (2B/4B/9B) and Qwen3-8B via full SFT on "Imp" (score-improving) trajectories, plus an offline preference-learning stage using KTO on Imp and Reg trajectories jointly, producing the Finch-{2,4,8,9}B model family.

Key results:

• Across 22 held-out tasks, Finch models surpass base counterparts by 10.22% on average (abstract); Table 3 reports average gains of +1.56% (2B), +3.40% (4B), +3.17% (8B), and +10.24% (9B), with the largest single-task gains on ahc058 (+290.59%) and Transaction (+74.30%).
• On competitive programming (Table 4), Finch-9B raises average score from 32.46 (Qwen3.5-9B) to 46.01, and Finch-4B raises it from 14.52 to 31.97.
• With KTO offline RL (Table 5), Finch-8B+KTO surpasses the best human score on both AC1 (1.5089 vs. 1.5097) and AC2 (0.9146 vs. 0.9015).
• Combined with test-time RL via nanodiscover (Table 6), Finch-8B matches state-of-the-art on two circle-packing tasks (CP n=26: 2.635983; CP n=32: 2.939573) and improves on the Erdős minimum-overlap problem by +3.2% over base Qwen3-8B (0.380948 vs. 0.380932/0.403585).
• Scaling training tasks in Finch Collection from 15 to 355 improves held-out performance by 14.1% on average (e.g., AC2 rises from 0.8801 to 0.9122, PRISM from 22.36 to 23.93).
• Trajectory filtering retained 156,731 of 172,997 trajectories (90.6%), removing 6,321 (3.7%) systematic errors, 1,575 unrecoverable/breakage cases, and 8,370 (5.0%) overly long examples.

Why it matters / caveats: EFT lets smaller open-source models internalize cross-task discovery skills that previously required proprietary frontier models or per-task test-time learning, and can compose strategies from unrelated domains (e.g., applying recommender-system or numerical-optimization techniques to competitive programming). Stated limitations: trajectories were collected/evaluated using only the OpenEvolve scaffold (unclear generalization to stronger scaffolds like EvoX), test-time RL synergy was shown only on mathematical tasks (not yet verified on tasks like kernel engineering), and the current framework is single-turn and text-only, not yet extended to multi-modal or multi-turn discovery settings.

Problem: Existing agent-skill improvement methods retain only the final optimized skill artifact and discard the decision history (diagnoses, rejected alternatives, evaluation evidence), so later agents cannot understand why prior revisions were made and may repeat obsolete or already-failed fixes.

Method: SkillHone is a harness that separates each development step into role-bounded optimization and evaluation subagent dispatches (Optimization Agent Team, Evaluation Agent Team, and a runtime dispatcher, formalized as M = (Topt, Teval, D)). It maintains two linked repositories — a skill repository (SKILL.md, scripts, references, templates) and a skill-evaluation repository (probes, validators, traces, redacted reports) — and records each step as a decision record ht = (qt, rt, et, ot) (diagnosis, revision, redacted evidence, outcome). Evaluation subagents run candidate skills and return only redacted reports (never unredacted probe targets/validators/traces) to optimization subagents, which propose revisions using this evidence plus prior decision history; it is compared against Skill-Creator (iterative synthesis) and Hermes-Agent-Self-Evolution/Hermes-SE (GEPA-style reflective optimization) as baselines, using Claude Opus 4.6 as the development-time controller and Qwen3.6-35B-A3B as the execution/evaluation backbone.

Key results:

• On GAIA and WebWalkerQA-EN (raw open-web setting), SkillHone reaches 64.6% avg. on GAIA and 66.4% avg. on WebWalkerQA-EN, beating the commercially-backed deep-research agent (curated search) by 15.8 and 3.2 points respectively.
• Within the raw open-web setting, SkillHone outperforms Skill-Creator by +20.5/+28.3 points and Hermes-SE by +14.2/+13.4 points on GAIA/WebWalkerQA-EN.
• Transfer to a different execution backbone (Claude Sonnet 4.6, no additional optimization): SkillHone reaches 72.4% on GAIA, which is 10.2/15.7/24.4 points above Hermes-SE/Existing-Skills/Skill-Creator respectively.
• Ablation: removing decision history drops GAIA/WebWalkerQA-EN by 13.4/10.9 points; removing role separation drops them by 6.4/5.3 points (full SkillHone: 64.6%/66.4%).
• Deployment study on 7 internal tool-mediated analysis scenarios: SkillHone improves 6 of 7 skills over seeded versions, with an average accuracy gain of 18.8 points (e.g., +30.0 pp counting, +26.3 pp aggregation, +25.0 pp structure parsing, +0.0 pp list filtering).

Why it matters / caveats: Persistent, auditable decision history (diagnoses, rejected revisions, evaluation evidence, outcomes) lets later agents continue skill evolution across sessions without re-deriving prior rationale, and the gains transfer across execution backbones, suggesting improvement in the skill procedure itself rather than model-specific fitting. Stated limitation: SkillHone currently evolves a single skill in isolation and does not address jointly evolving multiple interdependent skills with shared resources or overlapping failure modes.

Problem: Video world models lack effective memory mechanisms, causing inconsistent generated scenes over extended durations, and existing rule-based context frame retrieval methods fail to generalize in scenarios with scene occlusions and dynamic objects.

Method: MemLearner introduces learnable query tokens (Q tokens) that act as an information bridge between context tokens (C tokens) and predicted tokens (P tokens) within a latent video diffusion model built on a causal 3D VAE and Diffusion Transformer (DiT) backbone, rather than adding a separate scratch-trained context query module. It uses two efficiency strategies: Strategy #1 restricts C/Q/P token interaction to a small number of shallow "Query Layers" (5 of 28 total layers) while deeper "Generative Layers" process only Q/P tokens, and Strategy #2 prunes attention computation to only three essential patterns (Q attending to P, Q attending to C, and P attending to P and Q). The model is trained with a multi-dataset strategy using dedicated camera encoders per dataset type (rendered data, estimated-pose real data, and unannotated real data with zero camera parameters), on a newly collected Unreal Engine-rendered dataset (100 videos, 13 scenes, ~18,000 frames each, 16.7 hours total) plus SpatialVid and Sekai-real real-world datasets.

Key results:

• On the authors' occlusion/dynamic-object dataset (Tab. 2), MemLearner achieves GT Comp. PSNR 21.23 / LPIPS 0.2904 and Revisit Comp. PSNR 18.57 / LPIPS 0.3230, outperforming CaM (19.85/0.3475, 17.61/0.3934), VMem (19.59/0.3872, 17.30/0.4187), DFoT (16.98/0.4796), and FramePack (16.42/0.5104).
• The "Separate Module" ablation (from-scratch context query module) collapses to GT Comp. PSNR 9.16, confirming it fails to learn context conditioning and degrades to a text-to-video model.
• On the CaM dataset (no occlusions/dynamics, Tab. 3), MemLearner and CaM perform comparably (PSNR 20.35 vs 20.22 GT Comp.), but CaM degrades sharply on the harder occlusion/dynamic dataset while MemLearner remains robust.
• Ablation of Query Layer count (Tab. 6) shows performance saturates around 5 layers (PSNR 21.23) versus 1 layer (18.16) or 20 layers (21.37 but higher compute), with fps dropping from 0.61 (1 layer) to 0.36 (20 layers).
• Removing the Q-attends-to-P attention pattern causes significant degradation (GT Comp. PSNR drops from 21.23 to 17.27, LPIPS worsens from 0.2904 to 0.4657).
• Camera pose embeddings can be withheld from C and Q tokens with no significant performance drop (PSNR 21.17 vs 21.23), suggesting the model implicitly learns geometric correspondence.
• Training solely on real-world data fails to learn memory capability (Revisit Comp. PSNR 15.32, LPIPS 0.6019) versus CaM+Ours+Real mixture (PSNR 18.49, LPIPS 0.3251).
• Training setup: 1B-parameter text-to-video DiT (28 layers, 5 Query Layers), 640×352 resolution, 77 frames, causal 3D VAE with temporal compression ratio 4 (20 latent frames), trained for over 20,000 iterations, batch size 8, learning rate 5×10⁻⁵.

Why it matters / caveats: MemLearner demonstrates that leveraging a pretrained video generation model's own attention for context querying (rather than a separate trainable module) enables learnable, adaptive memory that generalizes to occlusions and dynamic objects better than hand-crafted retrieval rules. Stated limitations include: at the 1B model scale, generation quality degrades when more than five characters interact simultaneously in a scene, and the approach still relies on full context storage rather than context compression, which the authors note is an orthogonal future direction (along with summarization, updating, editing, and selective forgetting).

Problem: It is poorly understood whether procedural memory (reusable skills distilled from LLM agent trajectories) actually produces knowledge that transfers across tasks, professional roles, and model backbones, rather than merely overfitting to the context in which it was learned.

Method: The paper introduces AFTER, a benchmark of 382 realistic enterprise tasks spanning six professional roles (Data Engineers, Data Scientists, GenAI Engineers, Infrastructure Engineers, Project Managers, Software Engineers) and 22 procedural skills stored as versioned SKILL.md artifacts, with controlled splits for specificity (in-context gain) and generality (cross-task, cross-role, cross-model transfer). Evaluation uses E VOLUTION, a harness implementing a COLLECT–DIAGNOSE–REVISE–PROMOTE cycle for skill updates, tested with reflectors/frameworks including Codex, Hermes, Memento, MemP, and EvoSkill, across models such as GPT 5.4, GPT 5.4 Mini, DeepSeek V4 Flash, Nemotron 3 120B, Gemma 4 (31B/26B/E4B), Qwen 3.5 (397B/122B/35B/9B), and GPT-oss (120B/20B).

Key results:

• Static skills improve full-pass accuracy (M2) by +2.8 points on average; e.g., Gemma 4 E4B gains +14.2 points on GenAI tasks while GPT 5.4 gains only +3.1.
• A single LLM-guided refinement round (Hermes reflector via Evolution, using Codex) yields an additional +5.2-point average gain, with per-scale aggregate improvements of +3.7 to +6.7 points.
• Skills evolved from diverse multi-model traces (n=5) achieve 73.1% cross-model test accuracy, versus 36.0–59.4% for single-model trace sources — a gain of at least +13.7 points over the best single-model source.
• Cross-role transfer for the pdf skill shows in-role gains of +11.7 (PM) and +6.2 (DS), but cross-role transfer causes losses of -4.8 (PM→DS) to -7.5 (DS→PM).
• Token efficiency: evolved skills cut total token usage by 326k tokens (62%) for Claude and 48k tokens (16%) for Hermes on the Kafka Lag Anomaly Detection task compared to handcrafted skills.

Why it matters / caveats: The results suggest the central challenge for procedural memory is not accumulating more experience but extracting procedural structure that remains useful beyond its original environment; the authors note limitations including narrow benchmark coverage (technology-sector roles only, excluding healthcare/legal/scientific domains), a fixed trace budget per evolution run that may not reflect larger real-world trace pools, and reliance on automated pytest verification that misses qualities like code readability or robustness beyond the test suite.

Problem: Safety and fairness evaluation of speech-capable models remains largely English-centric and reliant on synthetic voices, leaving unclear how vulnerable these models are to unsafe or stereotypical requests across languages and under naturalistic (non-adversarial) spoken conditions.

Method: The authors first survey safety documentation across 38 speech models (Speech-LLMs and OmniLLMs). They then build RedVox, a multilingual (English, French, Italian, Spanish, German) benchmark grounded in two existing text resources—SHADES (stereotypes) and M-ALERT (safety)—adapted by 52 community participants into two request types: "Speech" (harmful content vocalized plus a textual follow-up) and "Audio" (harmful content in text only, paired with silence, noise-a, or noise-b distractor audio). Responses are scored on a severity scale (Safe, Safe by Accident, Controversial, Unsafe) via an LLM-as-a-judge (GPT-5.5) combining safety/fairness and relatedness labels, validated against human annotation (Gwet's AC1). Eight models are evaluated: Qwen2-Audio, Phi4-Multimodal, Voxtral, Qwen3-Omni, Gemma 4, Gemini-Flash-Lite 3.1, Gemini-Pro 3.1, and GPT-realtime-2.

Key results:

• Only 8% of surveyed models document any multilingual safety analysis (11 of 38 models report any safety evaluation at all).
• The full RedVox collection totals 6,118 unique entries (~10 hours of audio/speech); after consent filtering only 50% of participants agreed to public release, yielding a public subset of 3,414 entries and 26 unique voices (model-ranking Spearman's ρ = 0.98 between full and released data).
• Voxtral produces fully unsafe responses in ~24% of cases (highest among tested models); proprietary models keep unsafe rates ≤3.1% (Qwen3-Omni close behind at 3.4%).
• English shows the lowest unsafe rate (5.1%) versus 10.0% for non-English languages (a ~96% relative increase); Voxtral reaches up to 28% unsafe in Spanish/French, a +15% absolute increase over English.
• Speech input is the most vulnerable modality, with combined controversial+unsafe (C+U) rates reaching 10-44% across models; even non-speech audio (silence/noise) raises harmful responses up to +20% versus text-only for Voxtral.
• The LLM-as-judge achieves 0.94 Macro F1 on relatedness and outperforms Qwen3Guard on safety/fairness classification (0.89 binary, 0.79 ternary F1/accuracy).
• In the participant questionnaire, 61.5% of the 52 contributors reported discomfort releasing harmful voice recordings, 56.4% felt personally responsible for pronouncing harmful content (23.1% "probably"), and 43.6% feared their voice being identified with harmful material (17.9% "probably").

Why it matters / caveats: The findings show speech and multilingual safety vulnerabilities persist even without adversarial jailbreaking, and that spoken/audio input itself acts as a "stressor" that increases harmful outputs beyond text alone—an underreported risk given the paper's own survey showing near-total absence of multilingual safety documentation in current speech model releases. Stated limitations include coverage of only five high-resource Indo-European languages, a naturalistic (non-adversarial, single-turn) rather than jailbreak-style evaluation setting, and reduced statistical power for native-vs-non-native speaker comparisons due to the reduced (consent-filtered) public dataset.

Problem: Existing text-rich image data construction pipelines follow a static crawl–filter–freeze paradigm that discards rejected samples (which often contain useful failure signals like OCR errors and semantic mismatches), causing later construction rounds to repeat the same failure modes.

Method: DataEvolver is a self-evolving multi-agent framework with four cooperative agents: a Retriever that collects candidate samples via policy-conditioned query planning, a Verifier that assigns quality scores (perceptual image quality, OCR/text recognition quality, semantic consistency) and rejection causes, a Critic that summarizes round-level feedback into natural-language semantic feedback for policy revision, and a Generator that performs targeted synthetic completion for under-covered topic-subtopic regions. In the implementation, query generation/coverage planning uses Mistral-7B, the ExperienceLibrarian/SemanticFeedback/PromptPlanner agents use Qwen3.5-4B, targeted image synthesis uses Qwen-Image, and Qwen3-VL is used only for optional captioning (not verification); OCR quality is measured with PaddleOCR and semantic consistency with CLIP ViT-B/32.

Key results:

• At the 0.75M data scale on PixArt-α, DataEvolver improves OCR-F1 over the strongest baseline (MARIO) from 4.56 to 8.45 on TextScenesHQ (an 85.3% relative improvement) and from 6.71 to 9.08 on LongTextBench (35.3% relative improvement).
• On Show-o2 at 0.75M scale, OCR-F1 improves from 0.19 to 0.45 on TextScenesHQ and from 0.27 to 0.44 on LongTextBench.
• Ablations on PixArt-α at 0.1M scale: removing the Critic drops F1 from 1.78 to 1.01 (TextScenesHQ) and 2.16 to 0.90 (LongTextBench); removing the Generator drops F1 to 1.40 and 1.37 respectively.
• Enabling the Critic raises construction-time mean OCR confidence from 0.861 to 0.938, and raises the share of high-confidence (>0.90) samples from 29.1% to 81.1%.
• Using a stronger Critic backbone (Qwen3.5-35B vs. Qwen3.5-4B) improves OCR accuracy from 0.41 to 0.66 and OCR-F1 from 0.75 to 1.14 on TextScenesHQ at the 10k scale.
• At the 0.5M scale, DataEvolver achieves 96.97% category coverage and 2.45% tail coverage of a text-rich image taxonomy, versus 90.91%/1.76% for MARIO and 78.79%/0.61% for AnyWord.

Why it matters / caveats: The results show rejected samples can be converted into actionable feedback that improves data construction itself (not just downstream training), and the benefit transfers across two different downstream generators (PixArt-α and Show-o2). The authors note the scaling results are a trend rather than a strict scaling law (too few data points/repeated runs for formal fitting), the framework's effectiveness depends on the reliability of the fixed Verifier (noisy OCR/semantic/quality scoring could propagate into policy updates), and the work is scoped to text-rich image generation, with extension to broader multimodal domains left as future work.

Problem: Small language models (SLMs) receive far less research attention than LLMs despite being highly relevant, so the paper investigates how SLMs perform as the generation component of a Retrieval-Augmented Generation (RAG) system, specifically whether a RAG pipeline built on SLMs can run on-device without GPUs within a reasonable time.

Method: The authors build a Russian-language RAG evaluation benchmark of 500 samples drawn from five datasets (DaNetQA, SberQuAD, RuRAG Test Dataset, Grounded-RAG-QA-RU, and a proprietary lecture-QA dataset of 5,000 conference presentations), categorize questions into six types (Factoid, Reasoning, Evidence-based, Comparison, Experience-based, Instruction) using Qwen3-8B, and evaluate 17 candidate GGUF-format SLMs (1B–8B parameters, CPU-only, ≤16GB RAM) plus GPT-5-mini as a baseline in a "context" and "no-context" generation mode. Answer quality is scored via a multi-judge LLM-as-a-Judge framework using four metrics (Correctness, Answer Relevance, Context Relevance, Faithfulness), with judges themselves selected/validated using F1, Average Bad Score, Pearson correlation to consensus, and Intraclass Correlation Coefficient (ICC).

Key results:

• Three judges were selected for final evaluation — GPT-5-mini, Qwen3-8B, and GLM-4.7 — based on F1, correlation, and Average Bad Score, achieving an Intraclass Correlation Coefficient (ICC) of 0.96 across judges.
• Qwen3-8B-Q4KM achieved the best small-model scores (Correctness 0.72, Answer Relevance 0.87, Faithfulness 0.83) but with 339.3s latency; Qwen3-4B-Instruct-2507-Q5KM scored close behind (0.71 / 0.89 / 0.80) at 70.9s latency and was chosen for production due to its quality/latency trade-off.
• GPT-5-mini baseline scored 0.73 Correctness / 0.88 Answer Relevance / 0.89 Faithfulness with context, but dropped to 0.47 Correctness (0.86 Answer Relevance) in no-context mode, showing context is critical for accuracy.
• Weaker models such as Llama-2-7B-Chat-Q4KM (0.32 Correctness, 115.0s) and Vikhr-Llama-3.2-1B-Q5KM (0.42 Correctness, 34.3s) lagged well behind the top performers.
• The benchmark's question set averaged 8.72 tokens per question (3–27 range) and 41.62 tokens per golden answer (1–364 range); pairwise cosine similarity across dataset sources ranged 0.06–0.12, and average question-complexity rating (1–10 scale via Qwen3-8B) was 4.94, with question-answer alignment scored 7.012/10.

Why it matters / caveats: The results suggest selected SLMs can match or approach LLM-level RAG generation quality while running entirely on CPU, supporting fully on-device, GPU-free deployment for resource-constrained or privacy-sensitive applications. Stated limitations include: evaluation covers only the generation stage (not retrieval/embedding/ranking), a single uniform prompt was used across all models rather than model-specific tuning, the scope is restricted to RAG-oriented tasks, and all findings are Russian-language only with uncertain generalization to other languages.

Problem: LLMs systematically misrepresent their internal uncertainty (fail at "faithful calibration," i.e., aligning expressed confidence with intrinsic confidence), a metacognitive deficiency that undermines trustworthiness even in frontier models.

Method: The paper introduces reinforcement learning with metacognitive feedback (RLMF), which uses GRPO (with the Liu et al. de-normalized advantage Ag = ρg − ρ) and scales each completion's advantage by a metacognitive term Zg = 1 − (Fpred − Fgold)² that measures how accurately the model judges its own faithful-calibration performance; this is paired with metacognitive data selection (choosing high- and low-scoring self-assessed training examples from PopQA) and a two-stage pipeline where Stage 1 (RLMF) calibrates numerical sentence-level confidence scores and Stage 2 applies a rewriting protocol (using Gemini-2.5-Flash-Lite) to map scores to natural linguistic hedges. Experiments apply this to Qwen3 (1.7B, 4B, 8B) and Llama3.1-8B-Instruct, compared against MetaFaith (metacognitive prompting) and FUT (Faithful Uncertainty Tuning/SFT) baselines, and against frontier models Gemini-3.1-Pro, Gemini-3-Flash, and GPT-5.

Key results:

• RLMF achieves cMFG ≥ 0.80 across all settings (numerical and linguistic), yielding average gains of 29% over MetaFaith and 25% over FUT in cMFG across 10 tasks.
• RLMF outperforms standard RL by up to 63% and, per Table 1, raises Llama3.1-8B-Ins cMFG* from 0.77 (+RL) to 0.84 (+RLMF), and Qwen3-8B from 0.51 (+RL) to 0.83 (+RLMF).
• Small RLMF-tuned models outperform large proprietary LLMs with specialized prompting, showing average gains of 37% over GPT-5, 17% over Gemini-3.1-Pro, and 25% over Gemini-3-Flash.
• Metacognitive data selection outperforms random and active-learning selection (e.g., Qwen3-8B: 0.83 cMFG* vs. 0.76 random and 0.72 active learning, Table 3).
• Human evaluation of the rewriting stage shows absolute win rates over FUT of 98% (diversity), 98% (naturalness), 95% (helpfulness), and 96% (contextual suitability), with inter-annotator agreement of 0.93.
• Gains are achieved while preserving task accuracy and factual calibration (e.g., Brier Score), unlike MetaFaith/FUT which can degrade these.

Why it matters / caveats: RLMF is positioned as a general paradigm for using metacognitive self-judgment as an internal RL feedback signal that avoids the diminishing-returns/degradation problems reported for prior RL-with-internal-feedback (RLIF) methods, with implications for LLM self-improvement, alignment, and trustworthy uncertainty communication in high-stakes settings; the authors note improved self-assessment is not equivalent to full metacognitive awareness and that faithfulness gains are not a substitute for factual verification.

Problem: Autoregressive Transformer mesh generators produce clean artist-like topology but are orders of magnitude slower than parallel continuous generative models, while diffusion/flow-matching methods cannot be directly applied to meshes because mesh connectivity is inherently discrete and incompatible with continuous noise injection and denoising.

Method: The paper introduces a topology embedder (trained with a spacetime-distance-based edge reconstruction loss following SpaceMesh) that converts discrete vertex adjacency into continuous per-vertex embeddings, which are recoverable via spacetime distance thresholding. This frozen embedder's output is concatenated with vertex positions and normals into a joint continuous state z = [p, n, e], which is denoised in parallel by PolyFlow, a Flux-style DiT (12 double-stream blocks, 24 single-stream blocks, hidden size 768, NoPE) trained with flow matching and a channel-weighted velocity loss, conditioned on point-cloud features from a frozen Hunyuan3D-Omni point-cloud VAE encoder. At inference, an Euler ODE solver (50 steps) with an EMA copy of the model generates positions, normals, and topology embeddings simultaneously, followed by edge decoding via spacetime distance thresholding, 3-clique face extraction, and normal-guided winding correction.

Key results:

• On Toys4K, PolyFlow achieves CD 0.008 / HD 0.021, outperforming the strongest AR baseline BPT (CD 0.014 / HD 0.035) by 43% in CD and 40% in HD, and beating MeshAnythingV2 (0.132/0.280), FastMesh (0.130/0.271), and DeepMesh (0.016/0.039).
• PolyFlow has the lowest CD standard deviation (0.001) among all compared methods.
• At the topology embedder ablation, dimension d=32 achieves F1=0.9991 (Prec 0.9983, Rec 1.0000) for edge reconstruction, versus F1=0.6000 at d=8 and F1=0.9697 at d=16; d=64 gives only +0.0005 F1 but worse end-to-end HD (0.025 vs. 0.021).
• Inference at 4,000 vertices takes 5.88s total for PolyFlow versus 36.72s for FastMesh (6.2× speedup) and over 9 minutes (554.5s) for BPT (tens of times faster); post-processing overhead stays under 70ms at all scales.
• Trained on ~5 million meshes; topology embedder uses 32-dim embeddings (16 space + 16 time) with a 512-hidden, 12-layer, 8-head Transformer trained for 350k steps; flow model trained on 64 GPUs with batch size 1 per GPU.

Why it matters / caveats: PolyFlow removes the sequential-decoding bottleneck of AR mesh generation while matching or exceeding its topology quality, and it uniquely supports exact, explicit control over output vertex count (demonstrated from 250 to 3,000 vertices on a single conditioning input) — a capability unavailable in existing AR methods. The prior related approach SpaceMesh (on which the topology embedding is based) was noted as limited to ~2k vertices due to transformer memory costs, motivating PolyFlow's integrated design.

Problem: Materials science figures in scientific literature—most of which are compound figures with a single caption describing multiple sub-panels—remain locked away from AI systems because direct image-text pairing is unreliable and no large-scale, modality-diverse multimodal dataset exists for the field.

Method: MatMMExtract is an end-to-end pipeline that parses publisher XML (Elsevier and Springer) into ⟨image, caption, in-text reference⟩ triplets, uses a fine-tuned YOLO12-m detector (trained on the newly created MaterialScope dataset of 2,811 manually annotated figures) to decompose compound figures into sub-panels, and then prompts an LLM (Gemini 3.1 Flash Lite, selected after benchmarking six models) with the caption, reference sentence, and a curated two-level materials-science taxonomy (19 categories, 100+ subtypes) to generate grounded sub-captions, category/subtype labels, and summaries via a JSON-schema-constrained API call. A CLIP-style dual-encoder (CLIP ViT-B/32 image encoder + MatSciBERT text encoder, trained with symmetric InfoNCE loss and indexed with FAISS) is built as a downstream retrieval baseline on the resulting MatSciFig dataset.

Key results:

• MatSciFig contains 391,606 panel-level image-text pairs extracted from 180,571 figures across 14,810 open-access articles.
• Fine-tuned YOLO12-m achieves mAP50 of 0.9227 on the MaterialScope test split, beating Exsclaim (prior published method) by 13.3 percentage points on mAP50, and beating the next-best retrained model YOLO8-m (0.9028).
• Inter-annotator agreement on MaterialScope reached Cohen's Kappa κ = 0.9245 (near-perfect agreement) over 102 shared images.
• Among six benchmarked LLMs, Gemini 3.1 Flash Lite gave the best cost-quality trade-off: 82.0% of sub-captions and 83.5% of summaries rated "good," with the lowest hallucination rate of 4.8% (vs. 9.0% for Gemini 3.5 Flash and 15.4-30.2% for the other four models), at $0.25 per million input tokens.
• 62% of figures in the curated corpus were compound figures, consistent with the 40-60% range reported for biomedical literature.
• The dual-encoder retrieval baseline improved R@1 from 2.4%/1.7% (zero-shot CLIP, image→text/text→image) to 10.5%/9.2% (fine-tuned), a 4.4x and 5.4x improvement respectively; R@10 reached 38.6%/36.7% and R@100 reached 69.7%/68.7%, with gains observed across all 19 visualisation categories.

Why it matters / caveats: The work unlocks decades of experimental materials knowledge embedded in figures for vision-language model training, releasing MatMMExtract (as a PyPI package), MaterialScope, and MatSciFig openly; stated limitations include that some categories (e.g., Photograph, Simulation, fine-grained plot types like Scatter Plot/Bar Chart) remain hard for all benchmarked LLMs, quality labels were assigned by a single materials science PhD student, and absolute retrieval performance (e.g., R@1 around 9-10%) remains modest, indicating substantial room for future work such as richer fusion architectures.

Problem: Existing mobile GUI agents are trained and evaluated largely with offline success trajectories, simulated environments, and static benchmarks, which fail to capture the real-device state distribution (account states, permission dialogs, payment authentication, risk-control mechanisms) and thus leave a persistent gap between high benchmark scores and real-world usability.

Method: Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent built on the Qwen3-VL-30B-A3B-Instruct backbone, trained through a real-device-dominant hybrid infrastructure (physical devices as primary execution environment, sandboxes as auxiliary support) combined with an error-driven data flywheel (interactive annotation of first-key errors plus teacher-model scoring and bounded takeover to generate recovery trajectories). Training follows a three-stage curriculum: supervised fine-tuning (SFT), step-level reinforcement learning (Step RL) using Group Sequence Policy Optimization (GSPO) with a hierarchy-triggered cascade reward, and agentic reinforcement learning (Agentic RL) also using GSPO at the turn level with curriculum sampling based on smoothed success-rate estimates (following STEP). Evaluation uses public grounding/navigation benchmarks (ScreenSpot-V2, MMBench-GUI-L2, OSWorld-G/-Refine, AndroidWorld) plus a newly introduced real-device benchmark, RealMobile, with fine-grained sub-goal scoring, veto conditions, and dual XML+logical-semantic-rule verification.

Key results:

• Xiaomi-GUI-0 achieves a 72.0% success rate on RealMobile and 78.9% on AndroidWorld (exceeding UI-Venus-1.5-30B-A3B's 77.6%).
• On grounding benchmarks: 94.7% on ScreenSpot-V2, 82.7% on MMBench-GUI-L2, 58.7% on OSWorld-G, 64.2% on OSWorld-G-Refine.
• On RealMobile, it substantially outperforms open-source models such as MAI-UI-8B (33%) and exceeds several proprietary systems, including Gemini 3.1 Flash (58%), Claude Opus 4.7 (60%), and Claude Opus 4.6 (33%), while approaching frontier models Gemini 3.1 Pro (85%) and Seed 2.0 Pro (80%).
• Per-domain RealMobile results: 100.0% success on Foundation (matching top proprietary models), 43.8% on Safety & Reflection (highest among open-source models, though the weakest domain overall), 66.7% on Memory & Knowledge, and 80.5% on Complex Reasoning & Planning (vs. only 31.7% for the strongest open-source baseline).
• SFT used ~1.2 million GUI step-level samples from ~120 thousand trajectories plus 4.4 million grounding samples; Step RL added ~0.4 million samples from ~40 thousand trajectories; training ran on 64 NVIDIA H100 GPUs (8 nodes x 8 GPUs).
• Trajectory-level cleaning (via sub-task decomposition and VLM judging) agreed with human spot-checks 94% of the time on 320 stratified trajectories.

Why it matters / caveats: The results support the paper's central claim that real-device training and evaluation (rather than emulators/static benchmarks) is key to closing the benchmark-to-deployment gap, and show a deployable-scale model (30B-A3B) can rival much larger frontier proprietary systems on complex, long-horizon real-world tasks. The paper notes Safety & Reflection remains the weakest capability domain across all evaluated models (even top proprietary models reach only ~62.5%), indicating safety-aware, self-corrective behavior is a persistent bottleneck; it also notes Xiaomi-GUI-0 trails frontier models on Memory & Knowledge, suggesting knowledge-intensive recall may depend more on raw model capacity/scale.

Problem: Dense supervision methods for long-horizon LLM agents (which score intermediate actions rather than relying on sparse outcome-only rewards) are conventionally evaluated only by plugging them into a full downstream RL training pipeline, which is expensive, conflates signal quality with training engineering confounders, and makes different methodological families incomparable.

Method: The paper introduces QVAL, a training-free testbed that labels state-action pairs (s, a) with a reference Q-value Qπ(s, a) — the expected discounted return under a near-optimal reference policy π, estimated via a Max-Value Monte Carlo (MVMC) rollout strategy — and then measures "Q-alignment" of a candidate dense supervision method's score k(s,a) via Spearman's ρ (and Kendall's τ in the appendix) rank correlation against these labels. It is instantiated as QVAL-v1.0 across four environments (TerminalBench/TBLite, OpenApps, ALFWorld, FrozenLake) and evaluates 21 dense supervision methods grouped into seven families: Ranking, Direct (e.g., direct-single, direct-16, direct-batched, direct-sequential, GVL), Intrinsic scoring (∆Belief, LLM-as-a-Verifier), Self-Distillation (SDPO, SDPO-gt), Pre-trained (VIP, LIV variants), Embedding (VLM-RM, VLM-SOR variants), and Code generation (Eureka, Auto MC-Reward/codegen, codegen-avg), using six open-weight backbones (Qwen3.5 at 9B/27B/35B-A3B/122B-A10B and Gemma 4 at 26B-A4B/31B).

Key results:

• Over 1.2K evaluation experiments were run across 21 methods, 7 families, 4 environments, and 6 backbones.
• Simple direct-prompting and ranking methods achieve the highest Q-alignment on average, consistently outperforming more specialized families (Figure 2).
• The MVMC reference policy using GPT-5.5 (k=16 rollouts) reaches 100% Pass@16 on the TerminalBench (TBLite easy) subset, validating it as a strong continuation policy.
• Code-based methods showed the largest variance and degraded to negative correlations on TerminalBench, while direct-prompting methods stayed positive across all four environments.
• Environment dataset sizes: TerminalBench 118 trajectories/100 eval points, OpenApps 40 trajectories/94 eval points, ALFWorld 40 trajectories/100 eval points, FrozenLake 50 trajectories/100 eval points (Table 2).
• Method rankings were largely preserved when comparing text vs. image observation modality (though text produced stronger alignment overall) and when comparing Q-value vs. state-value V(s) reference targets, and when comparing TerminalBench labels generated by GPT-5.5 vs. Claude Opus 4.7 as the MVMC backbone (Figures 4 and 5).

Why it matters / caveats: QVAL provides a cheap, common-ground diagnostic to filter dense-supervision candidates by signal quality before committing to expensive post-training runs, and the authors explicitly note it is a filter rather than a replacement for downstream training evaluation, since factors like RL optimization/loss-integration choices and exploration-incentive signals that poorly align with Q-values could still be useful for training.

Problem: Existing brain encoding/decoding methods treat these as isolated, unimodal tasks that rely on external priors (e.g., frozen CLIP, diffusion models, LLMs) rather than exploiting the brain's intrinsic multimodal nature, so no unified framework achieves bidirectional mapping across brain, vision, and language.

Method: BrainJanus introduces a Unified Brain Tokenizer, a VQ-VAE-style tokenizer (codebook size 128, compression ratio 128, embedding dimension 32) trained from scratch with a reconstruction + codebook + commitment loss to quantize continuous fMRI signals into discrete tokens within a shared "Omni space" alongside vision tokens (from the image tokenizer of Sun et al., 2024) and text tokens (from the Chen et al., 2025 tokenizer). On top of this, an All-in-One autoregressive Transformer backbone (initialized from Janus-7B, hidden dimension 4096, fine-tuned with LoRA rank adapters on query/value projections) performs next-token prediction over interleaved multimodal token sequences, enabling four tasks: brain-to-image, brain-to-text, image-to-brain, and text-to-brain, trained jointly via supervised fine-tuning on the Natural Scenes Dataset (NSD).

Key results:

• Brain-to-text decoding: BrainJanus achieves BERTScore 38.12 and CLIP score 96.2%, surpassing prior state-of-the-art by 7.21 and 1.5% respectively (Table 2, Qwen-caption GT setting); under COCO-caption GT it reaches BLEU1 63.20, CIDEr 62.37, CLIP 94.8%.
• Brain-to-image decoding: BrainJanus attains 94.4% CLIP semantic similarity (highest among compared methods) despite diffusion-free generation, versus 93.0% for MindEye2 and 93.5% for UMBRAE (Table 3); zero-shot (trained only on brain-to-text pairs) variant still reaches 77.3% CLIP.
• Brain encoding (image/text to synthetic fMRI): BrainJanus reaches 72.8% Inception, 75.3% CLIP (image) and 77.0% CLIP, 28.3 BERT (text), outperforming Linear Regressive (65.9%/68.5%/72.5%/25.1) and Transformer Encoding (70.4%/72.4%/73.9%/26.4) baselines, versus an upper bound of 94.7%/94.4%/96.2%/38.1 using ground-truth voxels (Table 5).
• Evaluation-hacking analysis: a trivial "Padding Hacking" baseline achieves near-perfect scores (e.g., 100% Inception/CLIP, PixCorr 0.919) by leaking visual embeddings, exposing that standard reconstruction-based encoding metrics can be trivially gamed.
• Ablations show noise ceilings for encoding evaluation: inter-trial MSE floor ~0.55, and Cosine Similarity/Pearson Correlation upper bounds below 0.65; codebook size beyond 128–256 gives only limited gains in reconstruction/CLIP alignment.

Why it matters / caveats: BrainJanus is presented as the first unified autoregressive model bridging brain, vision, and language via a shared discrete token space, showing zero-shot cross-task generalization and biologically plausible/interpretable cortical topography in generated fMRI. Stated limitations: it is restricted to fMRI data from visual cortex rather than whole-brain activity, reliance on strong generative priors may cause "hallucinations" where visual quality is prioritized over biological faithfulness, and high computational cost plus generalization to more diverse neural modalities/subject populations remain unexplored.

Problem: Prior audio-video generation methods use separate per-modality tokenizers in a dual-branch architecture, which neglects the representation gap between the two modalities' embedding spaces (causing audio-visual semantic misalignment) and requires intensive computational resources for training.

Method: AVTok is a unified 1D tokenizer built on the LARP query-based transformer baseline, extended into a dual-stream transformer architecture with a shared encoder-decoder but modal-specific learnable holistic queries (video query set of size n=1024, audio query set of size q=128) and modal-specific normalization layers, producing 1152 holistic discrete tokens from a unified codebook. Training uses a hierarchical Video-First-Audio-Later (VFAL) strategy (video-only reconstruction in Stage 1, frozen-video audio-only training in Stage 2, joint decoder finetuning in Stage 3), a representation alignment loss (Lrep) that aligns patch embeddings with a pretrained audio-visual foundation model (CAV-MAE Sync), and a cross-modal AR generative prior (Lprior) adapted from LARP for two token orders (xv‖xa and xa‖xv) to support downstream Llama-like autoregressive generation for audio-to-video (A2V), video-to-audio (V2A), and class-conditional joint audio-video generation (cJAVG).

Key results:

• On VGGSound reconstruction, AVTok (1152 tokens) achieves PSNR 25.62 / rFVD 12.80 / LPIPS 0.126 for video and SI-SDR 23.09 / rFAD 5.93 / MR-STFT 1.523 for audio, outperforming its own vanilla single-stream variant (rFVD 14.87, rFAD 10.26) and video-only baseline LARP (rFVD 14.24) while remaining competitive with audio-only codecs like SpectralCodec (rFAD 5.56).
• For A2V generation, AVTok-A2V (208.4M tokenizer + 632.0M generator) reaches gFVD 150.26 vs. TempoTokens' 786.61 (83.7M/1.9B params).
• For V2A generation, AVTok-V2A achieves gFAD 49.47, better than SpecVQGAN (210.07), V-AURA (126.92), and VinTAGe (80.06), though MMAudio remains best at 17.09.
• For cJAVG, AVTok-cJAVG achieves gFVD 138.80, outperforming JavisDiT (1040.28, 8.9B params) and Ovi (972.65, 17.3B params), using far fewer parameters (208.4M tokenizer + 632.4M generator).
• Ablations show removing VFAL raises rFVD to 13.19/rFAD to 9.38 and hurts downstream gFVD/gFAD across all three tasks (e.g., cJAVG gFVD rises from 138.80 to 193.28); removing Lrep similarly degrades results (cJAVG gFVD 184.20); removing Lprior improves reconstruction (rFVD 10.63) but sharply worsens generation (cJAVG gFVD 249.47).

Why it matters / caveats: AVTok demonstrates the feasibility of jointly encoding audio and video into a single 1D latent space with a unified codebook, achieving competitive or superior generation quality with far fewer parameters than diffusion/flow-matching dual-branch baselines (e.g., 632M-class generator vs. 8.9B–17.3B for JavisDiT/Ovi), suggesting a path toward unified multimodal models for audio-video generation. Downstream generation experiments were limited to VGGSound only (not TAVGBench) "due to time and resource constraints," and no open-source baseline exists for the unified tokenization task itself, so comparisons rely on unimodal SOTA methods rather than direct competitors.

Problem: Generating high-resolution photomosaics — images that read as a coherent global scene from afar while every local tile is a convincing, self-contained image up close — is computationally expensive and existing methods cannot satisfy both scales simultaneously.

Method: PhotoQuilt is a training-free, model-agnostic framework built on pretrained text-to-image diffusion backbones (evaluated on Stable Diffusion 2.1, FLUX.1, and FLUX.2) that performs "bootstrapped tiled denoising": it first generates or encodes a coarse base latent at low resolution (via a text prompt c0 or a reference image I0), upsamples it to the target latent grid with a fixed upsampler, and re-injects noise once with an SDEdit-style partial renoising strength s to form a shared latent z̃s; each of K non-overlapping spatial tiles is then denoised independently from that shared latent down to t=0 under its own condition ck, and the tiles are decoded jointly into the final mosaic. It exposes two independent conditioning axes (global base condition and per-tile condition) supporting shared-prompt, per-tile-prompt, and image-gallery (via Redux for FLUX.1 or native image-to-image for FLUX.2) conditioning modes, and supports multi-GPU distributed generation via band-partitioned tile rows and block-tiled VAE decoding for ultra-high-resolution canvases.

Key results:

• On the 6144×6144 benchmark (768×768 tiles, 8×8 grid), PhotoQuilt (FLUX.1) achieves PSNR 30.15 / SSIM 0.89 / LPIPS 0.04 for global structure and CLIP 0.36 / BLIP 0.93 / Image Reward 0.21 for local tiles, outperforming all six baselines (Match & Tone, AdaIN, Color T2I-Adapter, NoiseBlend, StreamDiff, Phomosaic) on the combined global-local trade-off.
• Ablation on renoising strength: default s=0.6 balances both criteria, while s=0.2 maximizes global fidelity (PSNR 42.55, SSIM 0.99) but collapses tile quality (Image Reward −2.27), and s=0.8 favors tile quality (Image Reward 0.76) at the cost of global structure (SSIM drops to 0.62).
• Removing the bootstrap entirely ("Without Bootstrap") raises tile BLIP to 1.00 and Image Reward by +0.94, but global PSNR collapses to 13.10 and SSIM to 0.11, confirming the shared renoised base is the sole source of layout coherence.
• Bootstrap resolution ablation: shrinking the base to 256×256 collapses global structure (PSNR 9.88, SSIM 0.03), while 512×512 degrades mildly (PSNR −1.12, SSIM −0.02) relative to the default 768×768.
• Inference time at 6144×6144: PhotoQuilt (SD2.1) takes 97.15s and PhotoQuilt (FLUX.1) takes 209.59s, both faster than Phomosaic's SD2.1 implementation at 267.22s; Color T2I-Adapter is slowest at 527.34s.
• Demonstrated ultra-high-resolution generation up to 14336×14336 using multi-GPU distributed generation across 4×H100 GPUs, and 12288×6144 mosaics using real-image gallery conditioning.
• Extended evaluation at coarser/finer downsampling (32×32, 128×128, 256×256) shows PhotoQuilt (FLUX.1) achieves the best PSNR/SSIM/LPIPS at every scale, and PhotoQuilt (FLUX.2) leads HPSv2 and Image Reward at 128×128 and 256×256.

Why it matters / caveats: By confining attention to fixed tiles rather than using global attention, PhotoQuilt scales generation cost linearly (not quadratically) with canvas size, making arbitrary-resolution, training-free photomosaic generation practical across both U-Net and DiT backbones. A stated limitation is that in image-gallery conditioning mode, reconstruction quality depends on the diffusion backbone, since a denoised tile may diverge from its reference image.

Problem: Operating systems and desktop UIs are designed for human perception (pixels, icons, windows), forcing AI computer-use agents to rely on expensive, brittle screenshot/OCR-based perception with high token costs, visual ambiguity, latency, and coordinate uncertainty.

Method: LUMOS (Language-Model Unified Machine-Readable Operating-System Semantics) is a semantic interaction layer that converts native OS accessibility metadata (Microsoft UI Automation/UIA trees) and browser DOM/accessibility structures into compact, machine-readable "semantic blueprints" with stable element IDs, roles, names, values, bounds, and action affordances. It adds live semantic pointer grounding (ElementFromPoint-style UIA hit-testing) to identify the UI element under the cursor, and runs an LLM planner in an observe–plan–act loop that emits a single constrained JSON action from a small "Universal Action Schema" (observe, open_windows_search, open_app, click, double_click, drag, type_text, set_text, press_key, finish), validated by a schema/safety Guard layer and executed only through visible UI primitives. A Memory and Repair layer tracks prior actions/text to prevent repeated or append-style errors and stabilize handoffs like Windows Search queries.

Key results:

• The prototype's evaluation is largely a proposed plan (Section VIII) rather than completed benchmarks; the paper states "A full evaluation should test whether semantic operating-system grounding offers measurable advantages" via four proposed experiment families (vision vs. semantic grounding, blueprint compression, semantic pointer latency, multi-step desktop tasks).
• Diagnostic counts from Notepad debug logs (Fig. 6) show failure/repair signal counts of 22, 5, 4, 3, and 2 across categories: instruction copied instead of answered, append correction (e.g., "11+7 → 117"), repeated topic fragment, long prose weak newlines, and related repairs.
• Case studies demonstrate the architecture on two tasks: opening Notepad and writing generated text (32 native elements observed in the foreground window, text-entry target grounded to element ID "A2"), and a Windows Search handoff for an "outlook" query.
• The current prototype is validated only with regression tests (action schema coercion, generated-text handling, text replacement, Windows Search handoff, safety checks, blueprint refresh) — explicitly stated as not replacing a human-subject or benchmark evaluation.

Why it matters / caveats: LUMOS reframes existing accessibility APIs (originally built for screen readers/assistive tech) as a "cognitive infrastructure" plane for AI agents, suggesting a path toward AI-native operating systems with parallel human and agent interface planes. The authors explicitly caveat that the system does not claim human-level autonomy, is strongest only on simple text-entry/launch tasks, has not solved complex applications (video editors, mail clients, custom-rendered tools) end-to-end, depends on the quality of exposed UI semantics (incomplete accessibility trees, ambiguous/duplicate controls, custom-rendered surfaces are limitations), and no quantitative comparison against vision-based baselines has yet been performed.

Problem: Sheet music lacks a domain-specific foundation vision backbone, forcing optical music recognition (OMR) research to rely on brittle, task-specific systems trained on limited annotated data that generalize poorly to unfamiliar notation styles and engraving conventions.

Method: MuSViT (Music Score Vision Transformer) is a ViT encoder (12 Transformer layers, 768-dim embeddings, ~85M parameters, 2D sinusoidal positional encodings) pre-trained with Masked Autoencoders (MAE) using a two-stage curriculum: Stage 1 is a synthetic warm-up on DeepScoresV2 (512x512 crops, patch size 16, 50% masking ratio), and Stage 2 is real-world adaptation on 9.7 million full-page IMSLP scans (1024x1024 resolution, 70% masking ratio), with a lightweight decoder discarded after pre-training. A smaller variant, MuSViT-Light (384-dim, ~25M parameters), is also trained. The model is evaluated via linear probing and fine-tuning on four downstream tasks (full-page recognition, staff-level recognition, symbol detection via Faster R-CNN, and difficulty classification via MLP/GRU heads).

Key results:

• Full-page recognition (linear probing): MuSViT achieves 16.4% SER vs. 48.6–62.4% for general-purpose encoders (PaliGemma 2, Kosmos-2.5, Qwen3-VL, DINOv3-7B); fine-tuned MuSViT reaches 10.9% SER vs. 20.0% prior state of the art.
• Staff-level recognition: MuSViT gets 18.4% SER under linear probing (vs. 21.0–47.5% baselines) and 8.6% SER fine-tuned (vs. 8.0% SOTA).
• Music symbol detection (DeepScoresV2): 79.7% mAP / 80.7% w-mAP under linear probing (best among encoders, next best DINOv3-7B at 70.4%/62.0%); fine-tuned MuSViT reaches 97.0% mAP50 vs. 90.5% SOTA.
• Score difficulty classification: 47.4% Acc0 / 87.1% Acc1 under linear probing vs. 38.4%/84.3% SOTA; fine-tuned MuSViT reaches 54.2% Acc0 / 89.3% Acc1.
• Pre-trained on 9.7 million pages from ~400,000 distinct works from IMSLP.
• Embedding-transcription consistency: MuSViT shows Pearson/Spearman correlations of 0.606–0.714 between embedding distance and transcription distance, versus slightly negative correlations (-0.009 to -0.153) for all general-purpose encoders.
• Fine-tuned general-purpose baselines still underperform MuSViT while MuSViT is "5–82x smaller in parameters and 16–260x more efficient in GFLOPs."

Why it matters / caveats: The results establish that sheet music is a distinct visual domain requiring domain-specific pre-training, since general-purpose vision/vision-language encoders (regardless of scale) systematically fail to capture musical notation structure, and MuSViT's embedding space uniquely correlates with symbolic musical content. A stated limitation/design necessity is that single-stage MAE training directly on IMSLP causes dimensional collapse, so the synthetic warm-up stage is required for stable convergence.

Problem: Existing controllable satellite image generation methods convert native geospatial vector annotations (polygons, polylines, boxes, points) into rasterized or sparse formats, which either degrades precise geometric structure or requires prohibitively expensive dense annotation, and locks each architecture into a single supervision granularity.

Method: TerraDiT-Ω is a Latent Diffusion Transformer (initialized from TerraDiT-α, a flow-based generative model) that directly consumes native geospatial primitives via a Unified Primitive Encoder, which uses Fourier feature mappings and format-specific MLPs to embed polygons, polylines, bounding boxes, and points (with instance captions from a frozen LongCLIP encoder). Spatial grounding is achieved through Geometry-Aware Local Attention (GALA), which uses a MetaRBF+ module to predict rotated anisotropic Gaussian kernel parameters (σx, σy, θ) and a Spatial Geometry Field (signed distance fields for polygons/boxes, tubular distance fields for polylines) to multiplicatively modulate cross-attention between primitive and visual tokens; the model also incorporates geolocation conditioning via RANGE and representation alignment (REPA) with a satellite-specific DINOv3 encoder.

Key results:

• On Git-Rand-15k, TerraDiT-Ω-XL (T + Ω + L) achieves FID 9.25, sFID 4.38, LPIPS 0.3438, improving over TerraDiT-Σ-XL (FID 12.01) and TerraDiT-α-XL (FID 14.21).
• On Git-Dense-3.5k, TerraDiT-Ω (T + Ω + L) reaches FID 18.20 vs. TerraDiT-Σ-XL's 27.49, its largest margin over baselines.
• Classification Accuracy Score (CAS) on Git-Rand-15k: TerraDiT-Ω with full primitives (Ω) reaches Top-1 79.36 / Top-5 97.39, versus TerraDiT-Σ points (70.23/95.83) and an upper bound of 84.23/98.47.
• Ablations: removing primitive conditioning (Ω) degrades FID from 21.97 to 27.19; adding rotation (θ) and Spatial Geometry Field modulation improves FID from 24.08 to 21.97; GALA outperforms Cross Attention (FID 24.08), GSA (22.27), IMA (22.31), and ALA (23.52).
• Data augmentation gains: AID scene classification Top-1 accuracy jumps from 72.67 (no synthetic data) to 86.53 (×2 synthetic ratio); City-Scale road extraction APLS improves from 60.50 to 63.52 (+3.02); OpenEarthMap mIoU rises from 55.75 to 57.46; DIOR mAP@50 improves from 77.09 to 78.87.
• Annotating a single OpenEarthMap segmentation tile takes on average 2.5 hours, motivating the need for flexible annotation-budget conditioning.

Why it matters / caveats: The unified framework enables a single model to support controllable synthesis across annotation budgets (points to polygons) and to generate synthetic training data that improves four distinct downstream RS tasks (segmentation, detection, road extraction, classification) without task-specific architectures. The authors note that current geospatial datasets pair primitives only with standard OSM tags rather than rich visual descriptions, limiting fine-grained (color/texture) control, and they flag risks of misuse such as generating deceptive geographic data or enhancing harmful surveillance systems.

Problem: Existing spoken language models represent speech at a fixed frame rate (e.g., 25 or 12.5 Hz), which ignores the time-varying information density of speech and offers no ability to trade off quality for speed at inference time.

Method: FlexiSLM is a thinker-talker SLM built on a Qwen2.5-7B-Instruct backbone (Thinker) with a pretrained Qwen2.5-Omni audio encoder, a Frame Merging Module that adaptively compresses 25 Hz continuous features into a dynamic-rate sequence (≤12.5 Hz) using cosine-similarity-based frame merging, and a Talker Transformer that predicts dynamic-frame-rate FlexiCodec (FSQ-based) speech tokens plus associated frame-length tokens, decoded via a frozen flow-matching audio decoder and Vocos vocoder. It introduces "direct frame-rate control," conditioning the Talker on a sinusoidally-encoded target average frame rate (rather than an indirect merging threshold), and is trained in three stages (Talker pre-training, multi-task LoRA fine-tuning, full fine-tuning with Talker-to-Thinker connection) using a weighted cross-entropy loss over text, speech-code, and speech-length streams.

Key results:

• At 12.5/12.5 Hz, FlexiSLM-Stage3 achieves overall s2t/s2s scores of 72.4/67.2, beating Qwen2.5-Omni-7B (66.7/63.3) by 5.7/3.9 points, and outperforming Kimi-Audio-7B (69.7/57.2) and Mimo-Audio-7B (70.6/59.0).
• At 6.25/6.25 Hz, FlexiSLM reaches 70.2/64.3, still above all 7B baselines on s2s.
• Direct frame-rate control error is below 0.1 Hz across settings, versus threshold control's high variance (e.g., targeting ~8 Hz via τ=0.90 yields 3.91–10.74 Hz, σ≈0.70).
• Reducing output frame rate from 12.5 to 6.25 Hz nearly halves RTF (1.17 → 0.59), a ~2.7× speedup vs. Qwen2.5-Omni-7B at that setting (1.3× at matched 12.5 Hz), while s2s score drops only about 1.0 point (Stage3: 70.6/66.1 → ... /67.2 at 12.5/6.25).
• At more aggressive settings, s2t/s2s overall drops from 70.2/64.3 (6.25/6.25) to 69.0/60.4 (5.0/5.0) to 67.2/56.5 (4.0/4.0); LibriSpeech clean/other WER worsens from 2.55/6.37 to 3.34/7.85 to 4.47/9.53 over the same rates.
• Model sizes: Talker has 630M parameters, Audio Encoder 640M parameters, each Merging Transformer 20M parameters; trained on 24 A100 80G GPUs.
• Ablations: replacing dynamic output merging with uniform merging drops s2s Overall AVG from 63.0 to 61.0 and increases TTS WER by 59% relatively (3.11 → 4.95).

Why it matters / caveats: FlexiSLM is presented as the first SLM with dynamic and controllable frame rates on both speech input and output, letting a single deployed model span operating points from 12.5 Hz down to 4.0 Hz without retraining for flexible compute/quality tradeoffs. Stated limitations: it has not been combined with RLHF/DPO post-training, it is not a streaming/causal model, and its training data does not cover reasoning-intensive tasks, multi-turn dialogues, or many multiple-choice questions, limiting generalization.