Technical investigation · 15 February 2026

Why Agent Workloads OOM at 62% GPU Utilization: KV Cache, Multi-Turn State, and a $27k Weekend

Author: Abi Aryan
Role: AI Infrastructure Engineer
Focus: LLM inference, GPU engineering, distributed systems, LLMOps, Joule, inference power economics engine, Maven, vLLM, SGLang, KV cache
Published: Invalid Date
Type: Technical investigation
Topics: LLM inference, KV cache, inference observability, queueing

Abstract

This investigation reconstructs a weekend where an agent demo that looked 10× over-provisioned lost seven of twelve replicas and spent $27k. nvidia-smi still reported 62% GPU utilization. The limiting resource was not FLOPs. It was KV-cache residency across 50–70 turn agents with tool retries. The post separates stateless RAG from stateful agents and gives admission rules that would have stopped the bleed.

Key takeaways

GPU utilization is a vanity metric when the replica is dying of memory, not FLOPs.
Stateless RAG discards context. Agents lease HBM for the life of the thread.
KV cache plus fragmentation can make "free" memory unallocatable.
Price per solved task, failures included, not per happy-path token.
Cap context growth and timeout stuck agents before you add replicas.

The question

Why did an agent serving stack die at 62% GPU utilization and spend $27k over a weekend?

Definitions

KV cache: The key-value attention cache maintained during autoregressive inference. On an agent thread it is a lease on HBM that grows with turns and tool traces.
Stateless RAG: Retrieve, generate, discard. Context does not persist across decisions. Retry is cheap.
Agent thread: Plan, act, observe, repeat. State persists across tool calls. Retry is expensive because the cache is still warm and still resident.
HBM fragmentation: Memory that nvidia-smi reports free but the allocator cannot place as a contiguous KV block.
Goodput: Successful completed tasks inside the SLO. A burned replica is zero goodput at full rack cost.

Figure 1. The Friday capacity model assumed short, discardable context. Saturday users ran multi-turn agents.

System overview

Capacity model (Friday): Weight size plus a short context. Tokens per request. Dollars per million tokens. The spreadsheet said 10× headroom.

Workload (Saturday): 50–70 turn agents with tool retries. Context accumulates. KV cache stays resident.

Replica: One GPU (or TP group) running an inference engine. Dies when the allocator cannot place the next KV block.

Gateway: RelayServe is the one I use to see request IDs and metrics before the node is already dead.

What I observed

I expected 10× headroom. The Friday math said so.

Saturday, real users ran 50–70 turn agents with tool retries. Seven of twelve replicas died. nvidia-smi still said 62% utilized. The weekend bill crossed $27,000.

One long thread could spike a node from about 150 GB to 220+ GB of resident state. Fragmentation did the rest.

Why I think this happened

The question that matters is not “how much does one forward pass cost?” It is “does state persist across decisions?”

Stateless RAG: retrieve, generate, discard. Linear. Retry is cheap.

Agents: plan, act, observe, repeat. The KV cache is a lease on HBM. Utilization can look modest because decode is memory-bound and often leaves SMs idle while the allocator is already out of placeable cache.

Finding 1: 62% utilization is compatible with an OOM outage

Claim. A replica can be memory-dead while SM utilization looks healthy.

Evidence. Seven of twelve replicas died. The dashboard still read 62% GPU utilization. Resident state moved from ~150 GB to 220+ GB on long threads.

Explanation. Utilization counts busy SMs. OOM is an allocator event. Prefill/decode kernels can look relaxed while KV pages and fragmentation exhaust HBM.

Implication. Page on allocatable KV headroom and replica death, not on nvidia-smi.

Finding 2: Agent cost is a lease, not a token

Claim. Multi-turn agents change the unit of capacity from tokens to resident state × time.

Evidence. A back-of-envelope that treats KV as concurrency × kv_per_turn × turns jumps from a demo footprint to a Saturday footprint without changing the weights.

Explanation. Each tool retry extends the lease. Failures still occupy cache until timeout.

Implication. Price per solved task, failures included. Cap context growth. Timeout stuck agents. Refuse to scale on GPU util alone.

def agent_vram_gb(weights, kv_per_turn, turns, concurrency):
    # Weights are the easy part. The KV cache is the lease.
    return weights + concurrency * kv_per_turn * turns

print("demo", round(agent_vram_gb(145, 0.9, 8, 4), 1), "GB")
print("saturday", round(agent_vram_gb(145, 0.9, 70, 12), 1), "GB")

Scenario	Weight (GB)	Turns	Concurrency	Implied resident (GB)
Friday demo	145	8	4	173.8
Saturday traffic	145	70	12	901.0

The Saturday line is why twelve replicas were never twelve replicas. It is a napkin, not a profiler dump, the 150→220 GB observation is the measured spike on a single node; the table shows how fast the same rule explodes when turns and concurrency move together.

Practical guidance

Separate capacity models for RAG and agents.
Cap max turns and max cached tokens per thread.
Time out tool loops. A stuck agent is a cache leak with an API.
Export request IDs, KV bytes, and replica death from the gateway, RelayServe’s reason for existing.
Do not add GPUs because utilization is 62%.

Limitations

This is a reconstructed incident, not a public reproducible benchmark. Hardware SKU, exact model, and engine flags are not named here. The VRAM function is linear and ignores fragmentation, which makes the real Saturday worse than the formula. Other agent frameworks with aggressive cache eviction will fail differently, usually as collapsing TTFT rather than hard OOM.