Things you can run.

Joule

Founder. Joule is an inference power economics engine that ties physical GPU energy to token throughput and SLO goodput.

gpuengineering.com

The public list behind the domain: CUDA, kernels, serving engines, multi-GPU systems. ~600 stars as awesome-gpu-engineering.

RelayServe

Minimal LLM inference gateway for heterogeneous devices. OpenAI-compatible chat, batching, streaming, request IDs, /metrics. On PyPI as relayserve.

smol-vllm

Educational paged-attention engine: KV cache, continuous batching, preemption, prefill vs decode. Built to be read, not shipped as production vLLM.

fullstack-inferencing

vLLM on Lambda Cloud, FastAPI gateway, nginx, Prometheus, Grafana, optional Jaeger. Documents engine-flag limits, cost metrics, and Lambda image caveats.

AbideX

Zero-code OpenTelemetry monitoring for agent workflows (CrewAI, LangGraph, Pydantic AI). The Abide-era observability product.

awesome-observability

Curated tools and frameworks for LLM observability.

llm-cluster-simulator

Browser simulator for distributed LLM training and inference: memory, throughput, cost, parallelism.

ai-infra-fragility

Code for Cascading Fragility in National AI Infrastructure, market structure, exposure, network models. Submitting to AI & Society (Springer).

nexus-workshop-code

Packt Nexus: build a first agent / multi-agent system with CrewAI, roles, tasks, orchestration.

inferencing_maven

Labs for the Maven inference engineering cohort.

class-code

Class exercises that sit next to RelayServe and the serving labs.

concurrency

Concurrency exercises for inference workloads.

ray_project

Student exercise on concurrency and profiling with Ray.