# Abi Aryan

> Hi, I am Abi. I build inference systems for data centers, neoclouds, and hyperscalers. Founder of Joule. I teach on Maven.

Hi, I am Abi. I am an AI Infrastructure Engineer building inference systems for data centers, neoclouds, and hyperscalers. I write and teach inference engineering: serving path, KV cache, SLOs, GPUs, and the job around them.

## Joule
- [Joule](https://joule.lat/): Joule is an inference power economics engine that ties physical GPU energy to token throughput and SLO goodput.

## Teaching / Maven
- School: [https://maven.com/goabiaryan](https://maven.com/goabiaryan). 1.7K subscribers. amongst the top-most rated instructors on Maven.
- Invited Faculty for AI Inference Engineering at [Andreessen Horowitz Academy (The Academy SF)](https://theacademysf.com/).
- [AI Inference Engineering & Systems Design](https://maven.com/goabiaryan/inferencing) (8 weeks): A highly rated masterclass for mid-to-senior engineers. Build the stack by hand. Next live cohort 24 Oct - 19 Dec 2026.
- [AI Inference for FDE and PM Roles](https://maven.com/goabiaryan/inferencing-intro) (Inferencing 101): A masterclass on what is happening inside the model, for FDE, PM, and adjacent technical roles.
- Lightning lesson: [The Future of LLM Serving](https://maven.com/p/1b67c0/) (45 min)
- Lightning lesson: [How to do Observability right?](https://maven.com/p/d74587/) (60 min)
- Lightning lesson: [How much will my model cost? Napkin Maths 101](https://maven.com/p/f8237b/) (60 min)
- Lightning lesson: [What happens when you make an LLM call?](https://maven.com/p/52b7f0/) (45 min)
- Lightning lesson: [Whats next for AI Engineers?](https://maven.com/p/e10955/) (60 min)
- Lightning lesson: [AI Careers in 2025+: What No One's Telling You](https://maven.com/p/4ee6de/) (30 min)
- 1:1 mentorship: [https://topmate.io/goabiaryan/294674](https://topmate.io/goabiaryan/294674)

## The stack I teach
- 01 Model: Transformers, GGUF, Hugging Face
- 02 Framework: PyTorch, Accelerate
- 03 Compiler: Triton, TorchDynamo, CUDA graphs
- 04 Kernel: FlashAttention, CUTLASS
- 05 Memory: PagedAttention, Mooncake, LMCache, KV cache
- 06 GPU: CUDA, MIG, DRA, HAMi, DCGM
- 07 Communication: NCCL, NIXL, NVLink
- 08 Networking: InfiniBand, RDMA, Ethernet
- 09 Serving: vLLM, SGLang, TensorRT-LLM, llm-d, llama.cpp, RelayServe, LiteLLM, Portkey, OpenRouter
- 10 Scheduling: Kubernetes, EKS, GKE, Ray, Karpenter, KEDA, HPA, Gateway API
- 11 Latency / throughput: Prometheus, Grafana, Locust
- 12 $ / token: Lambda Cloud, Modal

## Doctoral research
- SLO-aware inference and hardware-aware compilers.
I am doing doctoral research in HPC distributed systems: SLO-aware inference optimization and hardware-aware compiler design. I am developing a digital twin that lets operators manage energy while protecting latency SLOs, via phase-aware observability, predictive forecasting, and human-centred decision support.
- SLO-aware inference optimization: Latency contracts first. Energy and cost moves are not allowed to blow TTFT, TPOT, or the tail.
- Hardware-aware compiler design: The compiler has to see the GPU in front of it: memory hierarchy, occupancy, and the serving loop, not a generic IR.
- Phase-aware observability: Prefill and decode are different workloads. The twin has to see which phase is burning watts and which is blowing the tail.

## Education and theses
- Now Doctoral research, HPC distributed systems: SLO-aware inference optimization, hardware-aware compiler design, and a digital twin for energy against latency SLOs.
- 2026 Master's in Information Management Systems, NOVA IMS: Thesis: Cascading Fragility in National AI Infrastructure.
- 2013-2014 MSc Applicable Mathematics, The London School of Economics and Political Science (LSE): Thesis: Pattern Recognition using Hopfield Neural Networks. Courses included algorithms and computation, game theory, cryptography, control optimization, non-linear dynamics, corporate finance, behavioral finance, and quantitative methods in finance.
- 2010-2013 BSc (Honors) Mathematics, Maharshi Dayanand University: Distinction (top 5). University record holder in Statistics. Pure and applied mathematics, with statistics, computer science, and operations research. Thesis: literature review on whether twin primes are infinite.
- Thesis 2026: Cascading Fragility in National AI Infrastructure. Master's thesis for the Master's in Information Management Systems at NOVA IMS. Submitting to AI & Society (Springer). Code on GitHub.
- Thesis 2014: Pattern Recognition using Hopfield Neural Networks. MSc Applicable Mathematics, The London School of Economics and Political Science (LSE), 2013-2014. Image pattern recognition using Hopfield models, energy-based recurrent neural networks.
- Thesis 2013: Literature Review on the Open Question in Mathematics: Are Twin Primes Infinite. BSc (Honors) Mathematics, Maharshi Dayanand University. Distinction (top 5). University record holder in Statistics.

## Books
- [GPU Engineering: AI Inference and System Design](https://abiaryan.com/books/): Packt. High-throughput inference serving from the gateway to bare metal: concurrency, multi-tenant isolation, control versus data plane, then fabric, silicon, compilers, and kernels. Almost finished. Launch is end of 2026 or early 2027.
- [LLMOps](https://www.oreilly.com/library/view/llmops/9781098154196/): O'Reilly book. Written 2023-2024, finished December 2024, published 2025. In English, now translated into Korean, Japanese, Russian, and Simplified Chinese.
- [What is LLMOps](https://www.oreilly.com/library/view/what-is-llmops/9781098154301/): O'Reilly report. Written 2023, published 2024. A separate brief from the LLMOps book.

## Pages
- [Home](https://abiaryan.com/)
- [About](https://abiaryan.com/about/)
- [Writing](https://abiaryan.com/writing/)
- [Books](https://abiaryan.com/books/)
- [Teaching](https://abiaryan.com/teaching/)
- [Advisory](https://abiaryan.com/advisory/)
- [Code](https://abiaryan.com/code/)
- [Speaking](https://abiaryan.com/speaking/)
- [llms.txt](https://abiaryan.com/llms.txt)
- [llms-full.txt](https://abiaryan.com/llms-full.txt)

- [Sitemap](https://abiaryan.com/sitemap/)

## Investigations
- [Why GPU Utilization Is a Misleading Metric for LLM Inference: Queueing, KV Cache, and SLOs](https://abiaryan.com/writing/eighty-busy-still-slow/): GPU utilization is not goodput. Queueing, KV pressure, and why $97 vs $0.38 / MTok. Keywords: GPU utilization, goodput, queueing, KV cache, SLOs, TTFT, TPOT, $/MTok.
- [Why KV Cache Needs a Directory: Locality, Leases, and Cache-Aware Routing](https://abiaryan.com/writing/kv-needs-a-phone-book/): On one GPU, KV is an allocator. Across a cluster it is a phone book. Keywords: KV cache, directory, locality, leases, cache-aware routing, allocator.
- [Why the Same GPU Serves One Request in 3.3 Seconds and Another in 39: Request Shape, Prefill, and Decode](https://abiaryan.com/writing/same-gpu-different-century/): Same model, same card, same code. The request shape did it. Keywords: request shape, prefill, decode, latency, same GPU.
- [One Inference Roadmap, Not Fifty Tools: Request Path Before CUDA](https://abiaryan.com/writing/one-roadmap/): Path, phases, memory, queue, then kernels. Not a bookmark pile. Keywords: inference roadmap, request path, CUDA, phases, memory, queue, kernels.
- [Why Agent Workloads OOM at 62% GPU Utilization: KV Cache, Multi-Turn State, and a $27k Weekend](https://abiaryan.com/writing/sunday-slack-27k/): The demo had headroom. Saturday agents leased HBM. Seven of twelve replicas died. Keywords: agent workloads, OOM, GPU utilization, KV cache, multi-turn, HBM, $27k.
- [The Inference Engineer Owns SLOs, Not vLLM Flags: Why This Job Didn't Exist in 2024](https://abiaryan.com/writing/job-that-didnt-exist/): The role owns TTFT, TPOT, KV pressure, and $/token when the demo becomes traffic. Keywords: inference engineer, SLOs, vLLM, TTFT, TPOT, KV pressure, $/token.
- [Junior Throws GPUs. Principal Deletes Them: Why Over-Engineering Hides the Queue](https://abiaryan.com/writing/junior-throws-gpus/): The funny ladder is a critique of architecture theater. Keywords: over-engineering, queue, GPU, architecture theater.
- [The Full Story of LLM Inference: Six Layers from HTTP Request to Multi-GPU Execution](https://abiaryan.com/writing/llm-inference/): The map: gateway, engine, orchestration, compiler, CUDA, silicon. Keywords: LLM inference, gateway, engine, orchestration, compiler, CUDA, silicon, multi-GPU.
- [Abi's PMPP Notes: Execution Model, Memory Hierarchy, and Why More Work Can Be Faster](https://abiaryan.com/writing/pmpp-notes/): Kirk & Hwu: warps, coalescing, occupancy, latency hiding. Keywords: PMPP, warps, coalescing, occupancy, latency hiding, memory hierarchy.

## Archive writing
- [Evaluation and Interpretation for LLMs](https://abiaryan.com/writing/llm-evals/): How evaluation changes when the model is generative.
- [Introduction to Large Language Models](https://abiaryan.com/writing/intro-llms/): What LLMs are, how they are trained, and the cost of building them.
- [Problems & Challenges in MLOps](https://abiaryan.com/writing/mlops-open-problems/): Open problems in taking models to production.
- [Problems & Challenges in AutoML](https://abiaryan.com/writing/automl-problems/): A 2019 survey of CASH, NAS, and autonomous model design.
- [Developing A.I. using Games](https://abiaryan.com/writing/game-ai/): Games as a testbed for machine intelligence.
- [Why TensorLayer?](https://abiaryan.com/writing/why-tensorlayer/): TensorLayer between TensorFlow and a usable deep learning API.
- [Dense Associative Memory for Pattern Recognition](https://abiaryan.com/writing/associative-memory/): Notes on Krotov and Hopfield, NIPS 2016.

## Projects I built
- [Joule](https://joule.lat/) (Company): Founder. Joule is an inference power economics engine that ties physical GPU energy to token throughput and SLO goodput. Keywords: Joule, joule.lat, inference power economics engine, physical GPU energy, token throughput, SLO goodput, phase-aware inference, power economics, digital twin.
- [gpuengineering.com](https://gpuengineering.com/) (Curriculum): The public list behind the domain: CUDA, kernels, serving engines, multi-GPU systems. ~600 stars as awesome-gpu-engineering. Keywords: gpuengineering.com, awesome-gpu-engineering, CUDA, kernels, serving engines, multi-GPU systems.
- [RelayServe](https://github.com/goabiaryan/RelayServe) (Serving): Minimal LLM inference gateway for heterogeneous devices. OpenAI-compatible chat, batching, streaming, request IDs, /metrics. On PyPI as relayserve. Keywords: RelayServe, relayserve, LLM inference gateway, heterogeneous devices, OpenAI-compatible, chat, batching, streaming, request IDs, /metrics, PyPI.
- [smol-vllm](https://github.com/goabiaryan/smol_vllm) (Engine): Educational paged-attention engine: KV cache, continuous batching, preemption, prefill vs decode. Built to be read, not shipped as production vLLM. Keywords: smol-vllm, smol_vllm, paged-attention, KV cache, continuous batching, preemption, prefill, decode, vLLM.
- [fullstack-inferencing](https://github.com/goabiaryan/fullstack-inferencing) (Lab / Lambda): vLLM on Lambda Cloud, FastAPI gateway, nginx, Prometheus, Grafana, optional Jaeger. Documents engine-flag limits, cost metrics, and Lambda image caveats. Keywords: fullstack-inferencing, vLLM, Lambda Cloud, FastAPI, nginx, Prometheus, Grafana, Jaeger, engine-flag limits, cost metrics.
- [AbideX](https://github.com/abide-ai/abidex) (Observability): Zero-code OpenTelemetry monitoring for agent workflows (CrewAI, LangGraph, Pydantic AI). The Abide-era observability product. Keywords: AbideX, OpenTelemetry, agent workflows, CrewAI, LangGraph, Pydantic AI, observability.
- [awesome-observability](https://github.com/goabiaryan/awesome-observability) (Observability): Curated tools and frameworks for LLM observability. Keywords: awesome-observability, LLM observability, tracing, metrics.
- [llm-cluster-simulator](https://github.com/goabiaryan/llm-cluster-simulator) (Systems): Browser simulator for distributed LLM training and inference: memory, throughput, cost, parallelism. Keywords: llm-cluster-simulator, distributed LLM, training, inference, memory, throughput, cost, parallelism.
- [ai-infra-fragility](https://github.com/goabiaryan/ai-infra-fragility) (Thesis): Code for Cascading Fragility in National AI Infrastructure, market structure, exposure, network models. Submitting to AI & Society (Springer). Keywords: ai-infra-fragility, Cascading Fragility, national AI infrastructure, market structure, exposure, network models, AI & Society.

## Course labs
- [nexus-workshop-code](https://github.com/goabiaryan/nexus-workshop-code) (Workshop): Packt Nexus: build a first agent / multi-agent system with CrewAI, roles, tasks, orchestration. Keywords: nexus-workshop-code, Packt Nexus, CrewAI, multi-agent, roles, tasks, orchestration.
- [inferencing_maven](https://github.com/goabiaryan/inferencing_maven) (Course): Labs for the Maven inference engineering cohort. Keywords: inferencing_maven, Maven, inference engineering, labs.
- [class-code](https://github.com/goabiaryan/class-code) (Course): Class exercises that sit next to RelayServe and the serving labs. Keywords: class-code, RelayServe, serving labs.
- [concurrency](https://github.com/goabiaryan/concurrency) (Course): Concurrency exercises for inference workloads. Keywords: concurrency, inference workloads.
- [ray_project](https://github.com/goabiaryan/ray_project) (Course): Student exercise on concurrency and profiling with Ray. Keywords: ray_project, Ray, concurrency, profiling.

## Elsewhere
- [ModelCraft](https://modelcraft.substack.com/)
- [GitHub](https://github.com/goabiaryan)
- [LinkedIn](https://www.linkedin.com/in/goabiaryan/)
- [X](https://twitter.com/GoAbiAryan)
- [YouTube](https://www.youtube.com/@goabiaryan)
- [Podcast](https://modelcraft.substack.com/podcast)
- [gpuengineering.com](https://gpuengineering.com/)