Writing
Investigations, then tools.
Investigations are the long pieces I write here. For these, I often start with a question, define the terms, make a claim, and look at what would break it.
Archive is earlier work: evals, AutoML, MLOps, games. ModelCraft is the newsletter and podcast. Field notes are the LinkedIn posts. Those two live on their own sites. I listed them here so you can find the source.
Investigations
- Why Your AI Factory Is Secretly a Dynamical System (And 5 Other Counter-Intuitive Truths About Modern AI Inference)
Power, cooling, KV, queues, and retries are one coupled system, not a dashboard of independent numbers.
- Why GPU Utilization Is a Misleading Metric for LLM Inference: Queueing, KV Cache, and SLOs
GPU utilization is not goodput. Queueing, KV pressure, and why $97 vs $0.38 / MTok.
- Why KV Cache Needs a Directory: Locality, Leases, and Cache-Aware Routing
On one GPU, KV is an allocator. Across a cluster it is a phone book.
- Why the Same GPU Serves One Request in 3.3 Seconds and Another in 39: Request Shape, Prefill, and Decode
Same model, same card, same code. The request shape did it.
- One Inference Roadmap, Not Fifty Tools: Request Path Before CUDA
Path, phases, memory, queue, then kernels. Not a bookmark pile.
- Why Agent Workloads OOM at 62% GPU Utilization: KV Cache, Multi-Turn State, and a $27k Weekend
The demo had headroom. Saturday agents leased HBM. Seven of twelve replicas died.
- The Inference Engineer Owns SLOs, Not vLLM Flags: Why This Job Didn't Exist in 2024
The role owns TTFT, TPOT, KV pressure, and $/token when the demo becomes traffic.
- Junior Throws GPUs. Principal Deletes Them: Why Over-Engineering Hides the Queue
The funny ladder is a critique of architecture theater.
- CASCADE: Why Fault Tolerance Without Adaptive Consistency Still Breaks GPU Clusters
Horovod, Ray, and DeepSpeed recover the job. They do not change how the cluster is allowed to disagree.
- The Full Story of LLM Inference: Six Layers from HTTP Request to Multi-GPU Execution
The map: gateway, engine, orchestration, compiler, CUDA, silicon.
- Abi's PMPP Notes: Execution Model, Memory Hierarchy, and Why More Work Can Be Faster
Kirk & Hwu: warps, coalescing, occupancy, latency hiding.
Archive
- Evaluation and Interpretation for LLMs
How evaluation changes when the model is generative.
- Introduction to Large Language Models
What LLMs are, how they are trained, and the cost of building them.
- Problems & Challenges in MLOps
Open problems in taking models to production.
- Problems & Challenges in AutoML
A 2019 survey of CASH, NAS, and autonomous model design.
- Developing A.I. using Games
Games as a testbed for machine intelligence.
- Why TensorLayer?
TensorLayer between TensorFlow and a usable deep learning API.
- Dense Associative Memory for Pattern Recognition
Notes on Krotov and Hopfield, NIPS 2016.
ModelCraft: Substack & Podcast
All issues →- Student Questions From my Classroom
2026-07-25 · 49 min. Questions from the Mewtwo inference-engineering cohort.
- The Biggest Mistake AI Engineers Make About Inference
2026-07-12 · From a talk in Hamel Husain's AI Product Engineering series.
- Fundamentals of Inferencing
2026-04-24 · 10 min. Hardware to throughput to kernels.
- Fundamentals of GPU Engineering
2026-03-26 · 40 min. Hardware–software co-design.
- The $27k Weekend: Why Your Agent OOMs at 62% GPU Utilization
2026-02-15 · Agents as stateful workflows on expensive hardware.
- The Real Reason LLMs Feel Slow
2026-02-12 · Prefill + decode latency. The mental model every AI engineer needs.
- Agentic AI's death is real BUT the Survivors are building Empires
2026-02-10 · From the Maven lightning talks.
- The Hidden GPU Crisis in AI Infrastructure
2025-11-04 · Most AI companies waste half their GPUs and don't know it.
- From Tensors to Teraflops: A Mental Model for GPU Engineering for LLMs
2025-09-27 · After the GPU Orchestration talk at Luma.
- Why the hell do I need to learn GPU Engineering for AI Systems?
2025-08-05 · The manifesto: AI engineers need the stack underneath the model.
- AI Careers in 2025+
2025-07-30 · From the Maven talk.
- How GPUs became relevant?
2025-04-12 · The breakthroughs that shrunk computers and supercharged AI.
- Decoding the Puzzle: Evaluation and Interpretation for LLMs
2023-06-14 · With Andy McMahon.
- Core Challenges in MLOps
2022-12-02 · Business, data, code, and model engineering across the lifecycle.
- An excellent year for software system design in ML
2022-11-03 · Favorite MLOps papers, compiled.
LinkedIn: Field Notes
LinkedIn · 50 →- The $27k weekend: why your agent OOMs at 62% GPU utilization
Sunday Slack huddle, 62% util, OOM. Agentic inference economics as an incident story.
- I ran the exact same model... 3.3 seconds vs 39 seconds
Same model, same GPU, same code. The request shape did it.
- The LLM inference pipeline nobody ever taught you
100+ component stack. Also the essay on this site.
- What is an inference engineer?
The emerging GPU / LLM performance role, and why it might be the one that matters in 2026.
- If I had to build a single skill roadmap to master LLM inference
A progression, not a pile of tools.
- >90% of people learning GPU engineering are doing it wrong
Execution model, memory model, occupancy, coalescing, divergence, reuse.
- The path is not PyTorch → CUDA → valuable. Wrong.
Understand the system first. Then pick the lever.
- The biggest fight in AI isn't more compute, it's memory
Prefill, decode, and why inference becomes a memory problem.
- Small vs large models require fundamentally different system design
Cross ~70B and the architecture changes. Most infra people can't do both.
- Junior vs mid vs senior vs principal AI systems engineer
Funny, and a critique of architecture over-engineering.
- If you can prompt or fine-tune an LLM but your production bills are exploding
From babysitting a model to inference engineering.
- The KV-cache directory problem nobody talks about
Generic inference education becomes a distributed-systems problem.
- NCCLX / Meta's extreme-scale LLM communication paper
GPU communication as the scaling limit.
- Why I'm betting on AMD EPYC over NVIDIA Vera Rubin
Heterogeneous infra, not NVIDIA-by-default.
- HipKittens, AMD GPUs, and the CUDA moat
Stanford Hazy Research, software that makes AMD GPUs actually fast.
- How I got into GPU engineering
Origin story: PMPP, compilers, electronics, RISC-V, tinkering.
- The full GPU engineering resource map
The pieces, in order. Then ModelCraft and gpuengineering.com.
- The Hidden GPU Crisis in AI Infrastructure
The organizing piece for the GPU sequence. ModelCraft.
- Why do I need to learn GPU engineering for AI systems?
The transition manifesto. ModelCraft.
- Why is everyone suddenly obsessed with kernels and CUDA?
Inference bottlenecks, memory layouts, datacenter economics.
- A mental model for GPU engineering for LLMs
The conceptual map before the tools. In the GPU-learning sequence.
- Concise notes on CUDA and GPU architecture fundamentals
Foundational notes from the same sequence.
- How GPUs became relevant
Historical context. Also on ModelCraft.
- GPU engineering skill roadmap
The larger educational sequence around mastering inference.
- The inference stack system diagram
The LinkedIn discovery surface. The durable map is on this site.
- The 80% GPU utilization trap
Latency explodes as utilization gets high. Queueing, not vibes.
- Why one team pays $97 and another $0.38 per million tokens
Inference economics made concrete.
- Prefill vs decode
Two different workloads. Tied to the 3.3s vs 39s experiment.
- KV cache and the memory hierarchy
HBM → CPU RAM → NVMe. Mooncake, LMCache, PagedAttention.
- GPU orchestration and fabric
NVLink, NCCL, RDMA, Karpenter, DCGM.
- Gateways and routers
LiteLLM, OpenRouter, Portkey, Envoy AI Gateway, SGLang Router, Ray Serve.
- Prefill vs decode pods / disaggregated serving
Distributed inference as topology, not a bigger box.
- Mooncake, Llumnix, Preble, DCQCN, Borg, Omega, Rail-only
What a serious inference engineer should read.
- vLLM block manager, NCCL, llm-d, Volcano, code to read
Inspect the code, not just the papers.
- What AI inference engineering and systems design actually covers
Course wrap-up: gateways, engines, KV cache, batching, profiling, hardware, concurrency, debugging.
- Inference engineering course / curriculum
Not just a theme. A curriculum people take.
- AI inferencing and system design patterns
The talk that packages the stack into one narrative.
- What's the difference between MLOps and LLMOps?
The bridge from the older identity to inference and GPUs.
- LLMOps book, early release
Author arc. Historically important.
- Fine-tuning LLMs, questions and tradeoffs
Earlier LLMOps / production-AI phase.
- How I got into GPU engineering
The origin story. Keep it.
- Causal Reflection with Language Models
NeurIPS 2025. Abide research, not only education.
- Abide's neuro-symbolic AI work
Reasoning, causal inference, agents, world models, before the GPU chapter.
- The LLMOps book going into copywriting
July 2025 release. More chapters on O'Reilly early release.
- LLMOps book early release
The more substantive book-introduction post.
- The calling / motivation post
From interesting work toward a sense of calling.
- LLMOps and software engineering: cohesion and coupling
Foundational engineering principles through the LLMOps lens.
- The fine-tuning LLMOps brief
LLMOps → systems → inference. The earlier layer.
- Inference reliability / Infer Summit
Reliability as latency, utilization, cost, and scale, not just uptime.
- Agent OOM / 62% GPU utilization incident
The flagship narrative. Slack + Zoom at 9:47pm.