distributed-inference
Everything on Ground Truth tagged “distributed-inference” — 1 item.
Mesh LLM lets you run models too big for any single machine by splitting them across peers News
Mesh LLM, the top project on Hacker News this week, runs models larger than any one machine can hold by partitioning them across networked peers -- layers 0-15 on one node, 16-31 on another -- over a serverless peer-to-peer transport, exposing a standard OpenAI-compatible API on localhost.