cuda
Everything on Ground Truth tagged “cuda” — 1 item.
A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type News
An open pull request to llama.cpp tracks which mixture-of-experts submodels get used most during inference and promotes them to GPU memory on the fly, roughly doubling decode speed on an 8GB card in the author's own tests - while slowing other models down.