News · 2026-07-24

Three separate tricks dropped the hardware floor for local AI in one day

Three unrelated local-inference projects landed on July 24, and together they attack three different bills. Noema's Overfit runtime ran a 26-billion-parameter Gemma model on an iPhone by keeping only about 1.4 GB resident and streaming roughly 13 GB of expert weights from storage. AMD shipped a 16-billion-parameter sparse model that activates 2.8 billion parameters per token. And CachyLLama, a llama.cpp fork, started saving conversation caches to SSD so agents stop reprocessing the same prompt.

Key facts

• The anchor number: Noema's Gemma 4 package holds about 1.4 GB of weights in memory and roughly 12.96 GB of expert pages on storage - a nearly ten-to-one split between what stays resident and what streams.
• When: all three surfaced July 24, 2026.
• Who: Noema AI, AMD's AGI group, and the independent CachyLLama project.
• Primary sources: the Noema Overfit package card, AMD's Instella-MoE model card, and the CachyLLama repository.

The headline that will travel is "26B model on a phone", and it needs unpacking, because the honest version is more interesting than the hype. Noema's Overfit runtime repackages compatible mixture-of-experts model files without any retraining. The shared, always-needed weights live in a resident file that stays in memory. The expert weights - 128 of them, of which eight are selected per token across 30 layers - live in page files on local storage and get streamed in and evicted on demand as the router picks them.

Think of it as the difference between owning a library and owning a library card. The phone is not holding 26 billion parameters in RAM; it is holding the index and fetching the shelves it needs. That works because a sparse model only ever touches a fraction of itself per token - the same property that makes these models cheap to serve in datacentres, repurposed as a memory trick.

What breaks is equally instructive. Cold prompts can touch experts broadly, so the first tokens hammer storage. Cache misses, storage speed, context length, memory pressure, thermals and how often experts get reused all determine whether it is usable. Noema says plainly that a smaller fully-resident model normally starts faster, generates faster, and uses less energy. The runtime is experimental, works only on selected mixture-of-experts models, and reverts to normal resident inference for some long-context and multimodal workflows. And no primary source publishes an actual token rate for this on an iPhone 17 Pro - Noema treats speed as device- and configuration-specific, and Google's published on-device figures cover only much smaller Gemma variants. Anyone quoting a phone token-per-second number for this is making it up.

AMD's contribution attacks a different bill entirely: active compute. Instella-MoE-16B-A3B-Think has 16 billion total parameters but activates 2.8 billion per token, across 27 layers with 64 routed experts plus two shared ones, selecting six routed experts per token. The genuinely novel piece is its FarSkip-Collective design, which restructures the dependency graph so expert-to-expert communication can overlap with computation using partial or slightly stale activations. That targets accelerator utilisation and distributed serving latency - a datacentre problem, not a phone problem. AMD says it trained on MI300X and MI325X GPUs through its ROCm stack, without disclosing GPU count or token budget.

Two caveats on the AMD release. The licence is ResearchRAIL, for academic and research use - not the Apache 2.0 the broader Instella family's open framing might lead you to expect. And the arXiv paper linked from the card resolves to the earlier Instella family report, not a mixture-of-experts technical report; AMD says that one is forthcoming. Current performance claims are AMD documentation, not paper-verified.

CachyLLama is the least glamorous and possibly the most immediately useful. It is an MIT-licensed llama.cpp fork that persists KV-cache checkpoints to SSD, keeps hot state in RAM, and restores unchanged conversation and system-prompt prefixes after a restart, with compatibility hashing, system-prompt sharing, and special handling for hybrid and recurrent architectures like Gemma 4 and the Qwen 3.5 and 3.6 families. The problem it solves is one every local agent user has felt: harnesses resend the full system instructions, tool schemas and history every single turn, and the model reprocesses all of it before producing a single new token. CachyLLama restores the matching prefix state and evaluates only the changed tail. Its own benchmark on an integrated-graphics chip reports long repeated agent prefixes going from minutes cold to about a second warm.

It is also refreshingly clear about what it is not. The project states it does not make decoding faster, does not add model support, and does not add quantisation types. Cache misses, changed prefixes, eviction or incompatible state all fall back to ordinary prefill. This is the local equivalent of prompt caching, which the hosted APIs have offered for a while.

The pattern is worth naming precisely, because the sloppy version of it is wrong. These are not one technology and they do not compose into "phones now run desktop models at desktop speed". They are three orthogonal savings: page only the experts you need (memory), activate only a sparse path (compute), and never reread an unchanged prompt (prefill). Each pays down a different bill. The honest caveat is that all three arrived as project announcements and self-reported benchmarks on the same day, with no independent reproduction, no comment threads, and no reviewed measurements yet.

Primary source, verified: read the paper →

Ground Truth. (2026, July 24). Three separate tricks dropped the hardware floor for local AI in one day. Ground Truth. https://groundtruth.day/news/three-ways-the-local-inference-floor-fell.html

@misc{groundtruth:three-ways-the-local-inference-floor-fell,
  title  = {Three separate tricks dropped the hardware floor for local AI in one day},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/three-ways-the-local-inference-floor-fell.html}
}

Topics: local-inference · mixture-of-experts · on-device · amd · kv-cache · efficiency · open-weight-models