AI Performance Engineering:
From GPU Kernels to LLM Inference

• GPUs began as game hardware built to render ~786,000 pixels per frame simultaneously — the same parallel structure neural networks need.
• AlexNet in 2012, trained on two GTX 580 GPUs, won ImageNet by 11 percentage points and launched the modern deep learning era.
• GPU performance grew 600× from 2012 to 2022: GTX 580 (1.6 FP32 TFLOP/s) → H100 (989 FP16 TFLOP/s) → B200 (4,500+ FP16 TFLOP/s).
• Tensor Cores (V100, 2017): FP16 is 8–16× faster than FP32. Dimensions must be multiples of 8 (FP16/BF16) or 16 (FP8/INT8) to avoid silent padding overhead.
• Compute grew 45× since 2017; memory bandwidth grew only 9×. Most AI workloads are memory-bandwidth-bound.

Sections: Why GPUs Exist · Fixed-Function to Programmable · Generation Timeline · Tensor Cores, HBM, NVLink · Memory: The Real Constraint

• Roofline Model: Pmax = min(C_peak, B_mem × I). Ridge point for A100 FP16 = 156 FLOPs/byte. LLM decode at batch=1 uses only 0.6% of A100 peak compute.
• Profiling hierarchy: nsys first (GPU idle?), torch.profiler second (which operator?), ncu third (why is this kernel slow?). Always profile before optimizing.
• Economics: $111/M tokens baseline → $28/M with continuous batching → $16/M with INT8 → $11/M with speculative decoding.
• Optimization order: algorithm (10–100×) → precision FP16/BF16 (1.5–8×) → operator fusion (1.5–3×) → memory layout (10–50%) → hardware scaling (last resort).

Sections: The AI Infrastructure Stack · The Roofline Model · Profiling Hierarchy · Economics of GPU Underutilization · Optimization Order

• CRITICAL: time.time() measures CPU submission (~0.05 ms), NOT GPU execution. Use torch.cuda.Event(enable_timing=True) + torch.cuda.synchronize().
• AMP (torch.autocast + GradScaler): runs FP16 forward/backward with FP32 master weights. 1.5–3× speedup. BF16 preferred on A100+.
• Calling .item() inside the training loop forces a GPU pipeline drain — can cut throughput by 50%+. Accumulate metrics on-device, sync once per N steps.
• Always warm up ≥5 iterations before timing — the first kernel launch triggers CUDA JIT compilation (100–1000 ms overhead).

Sections: Tensors · Autograd · nn.Module · The Complete Training Loop · GPU Timing — The Right Way · The 10 Most Common PyTorch Performance Mistakes

• PCIe 4.0 ×16 = 32 GB/s. H100 HBM3 = 3,350 GB/s — 100× mismatch. Every unnecessary .to('cuda') transfer burns PCIe bandwidth.
• pin_memory=True + non_blocking=True in DataLoader overlaps CPU-to-GPU copy with GPU compute — roughly 2× faster transfers.
• NUMA: cross-socket memory access adds 50–100 ns and 30–50% bandwidth penalty. Fix: numactl --cpunodebind=0 --membind=0.

Sections: CPU Architecture for AI Engineers · PCIe — The Bridge · GPU Hardware Fundamentals · Multi-GPU Configurations · Reading GPU Specifications

• Thread hierarchy: thread → warp (32 threads, SIMT) → thread block → grid. Warp divergence = serial execution of all branches.
• Memory coalescing: 32 threads accessing consecutive addresses = one HBM transaction. Non-coalesced = 32 transactions = 32× slower.
• Kernel fusion reduces HBM round trips: fuse LayerNorm + linear + activation to skip intermediate HBM writes.

Sections: GPU Thread Hierarchy · Memory Coalescing and the Memory Hierarchy · Tensor Cores · Kernel Fusion

• nsys: full system timeline (CPU + GPU + PCIe + NVLink). Long GPU-idle gaps = DataLoader or launch bottleneck. Run first.
• torch.profiler: CPU+CUDA time per PyTorch operator. Find the single slowest operator before changing any code.
• ncu: inside one kernel — memory throughput %, SM occupancy, Tensor Core utilization, warp stall reasons. Use third, only after identifying the slow operator.

Sections: Nsight Systems: System Timeline · Nsight Compute: Kernel Hardware Counters · torch.profiler: PyTorch Operator Attribution

• BF16 AMP on A100+: activates Tensor Cores fully with no overflow risk. 1.5–3× training speedup.
• INT8 quantization: 2× memory reduction, 1.5–2× speedup. GPTQ/AWQ for LLMs: 4× memory reduction (INT4) with minimal quality loss.
• torch.compile() fuses operators via Inductor. 1.5–4× speedup. Requires 2–3 warmup iterations. Use fullgraph=True for maximum optimization.

Sections: Numeric Precision and AMP · Quantization: INT8, INT4, FP8 · torch.compile · Batch Size Optimization

• Set num_workers=4–8, pin_memory=True, non_blocking=True, persistent_workers=True. These four settings eliminate most DataLoader bottlenecks.
• Diagnose: nsys timeline showing GPU-idle gaps between kernel bursts = DataLoader bottleneck. If GPU is never idle, DataLoader is not the problem.
• Advanced: NVIDIA DALI for GPU-side image decoding; WebDataset for streaming large datasets without small-file I/O overhead.

Sections: DataLoader Pipeline Tuning · Diagnosing I/O Bottlenecks · DALI · WebDataset

• perf stat: IPC < 1 = memory-bound or branch-heavy. LLC-load-misses > 20% = cache thrashing in the DataLoader CPU pipeline.
• CPU flamegraphs: width = time. Use perf record -g + Brendan Gregg's FlameGraph. Find CPU bottlenecks before tuning GPU code.
• eBPF/bpftrace: zero-overhead kernel tracing. Attach probes to syscalls, I/O, and scheduler events without recompiling.

Sections: perf stat — Hardware Event Counting · CPU Flamegraphs · eBPF and bpftrace · Lock Contention Analysis

• NUMA: GPU DMA is fastest with pinned memory on the same NUMA node as the GPU's PCIe slot. Mismatched NUMA reduces effective PCIe bandwidth.
• numactl --cpunodebind=0 --membind=0 python train.py: pins process to the same NUMA node as the GPU.

Sections: The CPU Memory Hierarchy · NUMA Architecture · Memory Bandwidth as a Bottleneck

• Two phases: Prefill processes all input tokens in parallel (compute-bound). Decode generates one token at a time (memory-bandwidth-bound, I ≈ 1 FLOPs/byte).
• KV cache: stores Key/Value tensors for processed tokens. LLaMA-2 70B FP16 at 4096-token context ≈ 8.6 GB KV cache per request.
• TTFT (Time To First Token) = prefill latency, critical for interactive apps. TPS (Tokens Per Second) = decode throughput, critical for batch jobs.

Sections: Prefill vs Decode Phases · KV Cache Memory Management · TTFT · TPS · E2E Latency

• Static batching pads to the longest sequence — 30–60% of compute wasted on padding tokens on typical distributions.
• Continuous batching (vLLM): new requests enter the batch as existing ones finish. GPU never idles. 4–8× throughput improvement over static batching.
• PagedAttention: KV cache in non-contiguous pages (like OS virtual memory). Eliminates fragmentation, enables near-100% VRAM utilization.

Sections: Static Batching and Padding Waste · Continuous Batching · PagedAttention · Disaggregated Prefill/Decode

• Small draft model (1–3B) generates k tokens. Large target model verifies all k in one parallel forward pass — much cheaper than k sequential decodes.
• At acceptance rate α=0.8, k=5 → ~3.6 accepted tokens/step → 3–4× speedup with zero change to output quality.

Sections: Draft and Target Model Mechanics · Acceptance Rate Analysis · Tree-Speculative Decoding

• Tensor Parallelism: split weight matrices across GPUs + AllReduce per layer. Requires NVLink — PCIe-based TP > 2 is impractical.
• FSDP/ZeRO: shards parameters, gradients, and optimizer state. Enables models larger than one GPU's VRAM.
• Amdahl's Law: 5% serial fraction caps max speedup at 20× regardless of GPU count.

Sections: Tensor Parallelism · NCCL Collectives · FSDP and Pipeline Parallelism · Mixture-of-Experts

• Five mistakes: no warmup, time.time() for GPU, single run (variance!), wrong batch size, ignoring thermal throttling.
• The throughput-latency curve 'knee' is max sustainable RPS before P99 explodes. Operate at 70–80% of that knee in production.
• Characterize arithmetic intensity, HBM bandwidth %, and SM occupancy BEFORE writing any optimization code.

Sections: The Five Benchmarking Mistakes · The Throughput-Latency Curve · Workload Characterization

• When porting a CPU workload to GPU, the bottleneck usually shifts from compute to memory bandwidth, then to DataLoader I/O.
• The porting checklist: profile first, move data to GPU once, eliminate Python loops, batch operations, use AMP, verify numerical correctness.

Sections: The Porting Checklist · Bottleneck Shift · DataLoader at Scale

• Optimization ladder: FP32 baseline → BF16 AMP (1.5–2×) → torch.compile (1.3–2×) → INT8 quantization (1.5–2×). Measure after each step.
• Target: 80–120 tokens/sec on a single A100 for a 7B model, vs 10–20 tok/s for an unoptimized FP32 baseline.

Sections: Establishing the Baseline · Precision and Compile Optimization Ladder · The Profiling Audit

• Pattern: GPU kernel bursts with long idle gaps in between. Nsight Systems makes this pattern immediately visible.
• Fix order: num_workers → pin_memory → persistent_workers → prefetch_factor → DALI → WebDataset.

Sections: Diagnosing I/O Starvation · Fixing with DALI and WebDataset

• Measure KV cache VRAM at context lengths 512 → 8192 tokens. KV cache grows linearly with context length × batch size.
• Memory budget: VRAM = model weights + KV cache + activations + framework overhead. Every component must be estimated before deployment.

Sections: Scaling the KV Cache · Memory Budget Planning

• Given a slow training loop, use the three-stage profiling hierarchy to isolate whether the bottleneck is in the DataLoader, the Python training loop, or the GPU kernel.

Sections: CPU-to-GPU Pipeline Diagnosis · Finding the Real Bottleneck

• B200: 5th-gen Tensor Cores with FP4, 192 GB HBM3e, ~8 TB/s bandwidth, ~4,500 FP16 TFLOP/s. NVL72 rack = 72 GPUs × 192 GB aggregate.
• Grace Blackwell Superchip: CPU + B200 connected by NVLink-C2C at 900 GB/s — eliminates PCIe bottleneck entirely for CPU-GPU transfers.
• FP4 precision: 2× memory reduction vs INT8. Quality retention depends on model type; quantize-aware training recommended for production.

Sections: 5th-Gen Tensor Core and FP4 · HBM3e: 192 GB, 8 TB/s · NVLink 5: 1.8 TB/s · Grace Blackwell Superchip · RTX 5090

• MI300X: 192 GB HBM3 (highest single-GPU VRAM available), 5.3 TB/s. Best for long-context LLM inference where KV cache size is the primary constraint.
• MI300X wins on memory-capacity-bound workloads; H100/B200 win on compute-bound (training, prefill at large batch sizes).
• ROCm 6.x: PyTorch works natively via CUDA compatibility layer. hipify-clang converts CUDA C++ to HIP with ~90% automation.

Sections: MI300X Architecture · ROCm 6.x Profiling · PyTorch on ROCm · MI300X vs H100 vs B200 Decision Guide

• Gaudi 3: 8× 200 Gbps RoCE ports on-chip, no separate NIC needed. Best for Transformer training at scale with mixed-precision workloads.
• Trainium 2 on AWS: 3× Trn2 instances are cheaper per training FLOP than GPU instances for long training runs at the same precision.
• Apple M4 Max: 410 GB/s unified memory bandwidth enables Q4-quantized 70B model inference on a laptop — impressive for local use cases.

Sections: Intel Gaudi 3 · AWS Trainium 2 · Apple M-Series AMX · CXL Memory Expansion

• Decision framework: memory-bound inference → MI300X or H100 80GB. Compute-bound training → H100 or B200. Cost-sensitive → Trainium 2 or Gaudi 3.
• Cost per token at scale beats cost per FLOP as the primary selection metric for production inference deployments.

Sections: Decision Framework · Cost per FLOP vs Cost per Token · When to Use Which Accelerator

• Triton (OpenAI): Python-based GPU kernel language. Higher-level than CUDA, comparable performance. FlashAttention-2 is written in Triton.
• Use Triton when you need a fused operation PyTorch doesn't provide natively, or custom attention patterns without writing raw CUDA C++.
• Triton compiles to PTX via LLVM. The compiler handles shared memory tiling, vectorization, and warp-level operations automatically.

Sections: Why Triton Exists · The Triton Programming Model · Vector Addition · Memory Coalescing · Fused GEMM

• Standard attention materializes the full N×N matrix in HBM. At N=4096: 64 MB per layer per head — read AND written every forward pass.
• FlashAttention tiles Q/K/V into SRAM blocks, never writes the N×N matrix to HBM. Memory complexity: O(N) instead of O(N²). 2–4× speedup for sequences >1024 tokens.
• FlashAttention v2 is default in HuggingFace Transformers 4.36+ via torch.nn.functional.scaled_dot_product_attention() for FP16/BF16 inputs.

Sections: Why Attention Is Memory-Bound · Tiling and SRAM Reuse · FlashAttention v1 and v2 · Profiling the Speedup

• DDP: each worker holds a full model copy; gradients averaged via AllReduce. Best for models that fit in one GPU's VRAM.
• FSDP: shards parameters across workers, gathers them for forward/backward, discards after. Enables training models larger than one GPU.
• Scaling laws: compute-optimal training (Chinchilla): C_opt ≈ 6 × N × D where N = parameters, D = training tokens. Most large models are undercomputed, not overcomputed.

Sections: Data Parallelism · FSDP vs DDP · Pipeline Parallelism · Scaling Laws

• DCGM: NVIDIA's production GPU telemetry. Exports SM utilization %, memory %, power, temperature, NVLink bandwidth to Prometheus.
• Key LLM SLOs: TTFT P99 < 2 s, TPS > 20 tok/s/user, availability > 99.9%. Burn rate alerting at 14.4× in 1 hour = page immediately.

Sections: Metrics · Logs · Traces · DCGM GPU Telemetry · Prometheus/Grafana · SLOs · Capacity Planning

• vLLM with PagedAttention and continuous batching: 4–8× throughput improvement over naive HuggingFace generate loop.
• Multi-LoRA: serve 100s of fine-tuned adapters from one base model deployment. Cost-per-token is the primary comparison metric across serving frameworks.
• Serving frameworks: vLLM (best throughput), TensorRT-LLM (fastest on H100), TGI (easiest setup), SGLang (structured output).

Sections: Deploying with vLLM · Throughput and Latency Under Load · Cost-Per-Token Analysis · Multi-LoRA Serving

• NVLink 4: 900 GB/s GPU-to-GPU. PCIe 4.0: 32 GB/s. For TP > 2, NVLink is not optional — PCIe kills efficiency.
• Measure scaling efficiency at 2, 4, 8 GPUs: if efficiency drops below 70% at 4 GPUs, find the serial fraction (Amdahl).

Sections: NVLink vs PCIe Scaling · Tensor Parallel Efficiency · The Scaling Efficiency Curve

• MFU (Model FLOPs Utilization) below 30% on a cloud GPU is a cost emergency. Every point of MFU improvement reduces your bill proportionally.
• Spot instances: 70–90% cheaper than on-demand for fault-tolerant training jobs with checkpointing. Use with automatic restart on preemption.

Sections: Reserved vs Spot vs On-Demand · GPU Utilization Economics · Right-Sizing · Cost-per-Token Optimization

• Top GPU questions: roofline model, Tensor Core alignment rules (multiples of 8/16), HBM vs GDDR6 bandwidth, SM occupancy, memory coalescing, warp divergence.
• Top LLM inference questions: prefill vs decode phases, KV cache memory formula, continuous batching benefit, speculative decoding acceptance rate math.
• Architecture design: serve 70B at 100 req/s with P99 TTFT < 2 s; FSDP vs DDP tradeoffs; KV cache sizing for given context length and batch size.

Sections: GPU & Systems Fundamentals · LLM Inference Questions · Profiling & Debugging · Architecture Design · 50 Practice Questions