System overview
Single-GPU KV allocator: Pages, blocks, eviction. PagedAttention lives here. smol-vLLM exists so you can read this loop without a 200k-line engine.
Cluster KV directory: Which node holds this prefix. Which tier. How stale. What a transfer costs.
Router: If it is hash(user) and the cache is sticky by accident, you will copy prefixes forever.
What I observed
On one GPU, KV is an allocator. Across a cluster, KV is a directory: who has this prefix, how stale is it, what does it cost to move.
Mooncake, LMCache, cache-aware routing, that is not “ML.” That is naming, locality, and leases.
Finding 1: Location turns cache into a distributed system
Claim. Generic inference education becomes a distributed-systems problem the moment cache has a location.
Evidence. The literature that matters here is prefix-sharing systems and cluster schedulers (Mooncake, LMCache, Llumnix, Preble), not another kernel blog.
Explanation. Reuse requires identity. Identity requires a phone book. Movement has a cost that can exceed regeneration.
Implication. Build or buy a directory before you buy another eight-pack to “add cache.”
# Once KV leaves a single GPU, you have a directory problem.
class KVDirectory:
def locate(self, prefix_hash):
# which node, which GPU, which tier (HBM / DRAM / NVMe)?
...
def pin(self, prefix_hash, locality_hint):
# routing without this is a coin flip
...Practical guidance
- Give prefixes an identity (hash of the token span, not the user id).
- Route on that identity. Log hit, miss, and migration cost.
- Treat HBM / DRAM / NVMe as tiers with explicit admission, not as “overflow.”
- Read the cache papers before another CUDA tutorial.
Limitations
This is a design argument, not a bake-off of Mooncake vs LMCache vs llm-d. Hit rates are workload-specific (shared system prompts vs unique RAG contexts). A directory can become the bottleneck if you design it like a chatty control plane.