kv-cache
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
Three separate tricks dropped the hardware floor for local AI in one day News
A 26-billion-parameter Gemma model ran on an iPhone by streaming expert weights from storage, AMD shipped a sparse mixture-of-experts model activating 2.8 billion parameters per token, and a llama.cpp fork began saving conversation caches to disk - three unrelated attacks on three different bottlenecks.
Grafting a Verified Solution Cache Lets a Frozen Model Skip Fine-Tuning Entirely News
A new paper shows a frozen small language model's accuracy on a hard math test can jump from 80.0% to 93.3% simply by grafting a byte-exact cache of verified solutions into it, with no weight changes at all.
Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies Lesson
Prompt caching lets an AI provider store the processed form of a repeated chunk of text -- like a long system prompt -- so it can be reused across requests at a fraction of the cost, instead of being re-processed every time.
New methods page an AI's memory in and out like an operating system News
Two new techniques treat a language model's long-context memory like an operating system's memory hierarchy - keeping coarse summaries on the GPU and paging compressed detail out to the CPU - with one, SeKV, cutting GPU memory use by 53% at 128,000 tokens while avoiding the errors that plague simpler methods.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
CachyLLama Tool
MIT-licensed llama.cpp fork that saves conversation and system-prompt caches to SSD and restores them after a restart, so local agents stop reprocessing the same prompt prefix every turn. Its own benchmark reports long repeated agent prefixes going from minutes cold to about a second warm.