AI Infrastructure Engineering:
Designing, Building, and Operating GPU Clusters, High-Speed Networks, Distributed Training, and LLM Inference at Scale

• This concise edition condenses the full edition’s 24 chapters to about a third of their length, keeping each chapter’s core mechanisms, key numbers, worked example, troubleshooting scenario and operating notes.
• The book is complete as a text; the practice material lives in the free companion repository github.com/sbittla/network-compute-engineer-ai-infra: the full-length text of every chapter, a lab for every chapter, 484 exercises with expected outputs, worked solutions and marking schemes, single-machine hands-on labs, and eight capstone projects.
• Every quantitative claim carries an evidence label: MEASURED, MODELED, SIMULATED, ILLUSTRATIVE, VENDOR-SPEC (dated) or OBSERVED (dated).
• ORION, an illustrative H100-class 256-GPU reference cluster run by the fictional Halcyon AI, is the running example so numbers can be traced from one layer to the next.
• The diagnostic loop: observe a symptom, collect evidence, trace the bottleneck across compute, network, storage and orchestration, identify the root cause, make a safe change, and validate with repeatable measurements.
• Labs come in three tiers: L1 runs on a laptop or paper, L2 on one GPU host (with a CPU fallback), L3 on several GPU nodes with an RDMA fabric. Everything needed to complete the course runs at L1.
• Reading paths are given for GPU or compute engineers, network engineers, ML or training engineers, SRE and operations engineers, and infrastructure architects, plus a plan for your first week.

Sections: About This Edition · How to Read This Book · Using the Practice Material

A 256-GPU training cluster and a 256-server web tier look alike but are different machines. The web tier’s servers are independent. A large training job turns the whole cluster into one synchronous computer that runs only as fast as its slowest GPU, network interface card (NIC), link, or storage target. An inference fleet sits in between: many independent requests, each served by a tightly coupled GPU group and…

• An AI training cluster is one synchronous computer, set by its slowest and least reliable component.
• Training is throughput under synchronization; inference is latency under load.
• Scale-up has 9× the per-GPU bandwidth of scale-out on ORION; TP traffic, 157× DP traffic, belongs inside the NVLink domain.
• The ceiling model classifies workloads; a fraction above 1 means bad data, well below ≈ 0.25 means latency or overhead.
• H100-class machine balance is ≈ 295 FLOP/B; BF16 decode at 2k context cannot reach it at any batch.
• 1 % of goodput is ≈ $67k/year at $3.00/GPU-hour [MODELED]; job MTBF is 48 h and checkpoints plus restarts cost ≈ 1.7 % of wall time [MODELED].

Sections: Why AI infrastructure is different · Training and inference · Reference architectures · Scale-up and scale-out · Characterizing a workload · Utilization, failure, and economics · The employer and role landscape

A 256-GPU cluster is not a pile of servers; it is a building-scale machine. Its throughput is set not only by GPUs and fabrics but by how many kilowatts a rack feed can deliver, how many litres per minute of coolant reach each cold plate, how much a floor tile can carry, and which single component can stop every job at once. This chapter takes the ORION reference cluster apart from the GPU to the utility feed: we…

• GPU racks are power- and cooling-limited; a BOM counts link ends and optics, not just links.
• Usable circuit power is √3 × V × I × derate × power factor; with 2N feeds one side carries the whole rack (45.1 kW needed, 65.55 kW available); check perfeed PSU capacity.
• Synchronized training swings ORION by ≈ 109 kW AC per step [ILLUSTRATIVE]; capping and power floors are the levers, and their cost must be measured.
• Q = ṁ · cp · ΔT sizes air (≈ 4,800 CFM all-air) and liquid (46.4 L/min of PG25 at 10 K).
• Whole-system dependency modeling finds SPOFs that no single drawing shows.

Sections: From a GPU to a building · The GPU server: where the watts go · Racks and the three networks · The power chain · Dynamic power, capping, and throttling · Cooling: moving heat out of the building · Facility interfaces and failure domains

A GPU server is still a Linux computer. Before a single CUDA kernel runs, the host has to read training data from disk or the network, place it in the right memory, copy it across PCI Express (PCIe), launch work on eight GPUs, service interrupts from ten network interface cards (NICs), and write checkpoints without stalling. When any of those steps is slow on one node, synchronous training makes the whole cluster…

• One slow host slows a synchronous job: a 2.6 s straggler costs ≈ 799 GPUhours per day on ORION.
• Read the topology rather than assuming it; discard since-boot samples and look per CPU.
• Utilization measures time, not work: acpi-cpufreq + powersave makes a busy core four times slower.
• %iowait is diluted and %util misleads on NVMe; NUMA-remote memory lowers IPC, not utilization.
• PCIe width below capability is always a fault; low speed on an idle GPU is usually power management.
• A collector must be read-only by construction, never escalate, and record every gap.

Sections: Why the host still matters · The HX8 host as Linux sees it · CPU scheduling and frequency · Memory: placement, pinned buffers, and errors · Storage I/O · PCIe: link training, bandwidth, and errors · Interrupts, the network stack, and the kernel log · The first ten minutes, and a safe collector

Most of the expensive mistakes in operating an AI cluster come from misreading what a GPU is doing. A dashboard says “GPU utilization 97 %” while the job runs at half its expected speed; a training step slows by 30 % after an innocuous library upgrade; a kernel that “should” take 80 µs takes 280 µs. None of these can be diagnosed from the outside. This chapter gives you a working model of the GPU execution machine…

• A GPU is fast only while it has queued work, resident data and enough parallelism per kernel.
• Short kernels make the GPU launch-bound; implicit synchronizations drain the queue (set_sync_debug_mode finds them).
• Occupancy matters only for kernels far below their roof.
• GPU-Util is a duty cycle: judge efficiency with SM-active, tensor-active and timelines.
• Rank fixes by modeled saving per step and validate each with repeated runs.

Sections: The GPU as a throughput machine · The execution model and latency hiding · The memory hierarchy · Launches, streams and synchronization · Occupancy: calculating it, and knowing when it matters · Tensor Cores, precision and batch size · Measuring: GPU-Util, DCGM and the profilers · Reading a timeline: five pathologies

Inside an 8-GPU server, bytes travel over NVLink and NVSwitch between GPUs, PCI Express (PCIe) between GPUs, network interface cards (NICs) and CPUs, and the inter-socket link. Which one a byte crosses depends on slots, cables, kernel options and how a job maps ranks to GPUs and NICs. Get it right and each GPU drives its own 400 Gb/s NIC through one PCIe switch; get one card or cable wrong and a large multi-node job…

• Collectives run at the speed of their slowest path; one misplaced NIC slows every rank.
• Gen5 x16 (63 GB/s per direction) just fits a 400 Gb/s NIC; x8 does not.
• Read nvidia-smi topo -m GPU rows against NIC columns through the legend; HCA index ≠ rail.
• Keep a per-node GPU→NIC/NUMA/slot/rail mapping and diff it against golden after every service.
• GDR needs platform P2P, peer memory, compatible versions and correct IOMMU/ACS, and proof (/GDRDMA).
• A golden topology can still hide AER storms, DevCtl drift, a small BAR1 or NVLink errors; read counters under load.
• Two bad nodes in 16 halve the health check busbw and double the job’s DP allreduce time [MODELED]; drain, checkpoint, requeue, repair, validate.

Sections: Why topology is a performance and reliability problem · PCIe: the host’s packet network · NVLink and NVSwitch: the scale-up network · Data paths: GPU↔GPU and GPU↔NIC · Topology discovery and the mapping artifact · GPUDirect RDMA prerequisites: verify, never assume · Multi-rail assignment as a joint placement problem · A labelled model of aligned versus misaligned all-reduce

“It got 3 % faster” is the most expensive sentence in AI infrastructure when it is wrong. Teams ship changes on one before/after number, then chase a regression that was really a hot chassis or a different PyTorch build. This chapter is about knowing what a GPU should achieve, through arithmetic intensity, the roofline and honest utilization metrics, and proving what it does achieve, through benchmark protocols and…

• For a BF16 linear layer, arithmetic intensity in FLOP/B is roughly the batch’s token count.
• ORION’s ridges are ≈ 295 (BF16), ≈ 591 (FP8) and ≈ 20 (FP32) FLOP/B; judge kernels against their binding roof.
• A point above a roof is a measurement error, not a fast kernel.
• W-TRAIN at 17.4 s/step is 40 % MFU, 53 % HFU with full recomputation; always state the convention and dense peak.
• Set targets against measured as well as vendor roofs, and name the clock each compute peak assumes.
• Record clocks_event_reasons; when the change alters the kernel, the clock is an outcome, so lock clocks or use a per-arm peer clock.
• Replicate with runs, not iterations; one pre-registered statistic drives the change, the bootstrap band and the exact permutation test, judged against an A/A floor and a practical threshold.

Sections: Performance engineering as a discipline · Compute limits, memory limits and arithmetic intensity · The roofline model on ORION · Utilization metrics: MFU, HFU and the ones that mislead · Clocks, power and thermal state · Designing a benchmark protocol · Statistics without hand-waving

In a 256-GPU training job the slowest link sets the pace for every GPU. The network is built from ordinary data-center parts: Ethernet, Internet Protocol (IP) subnets, the Border Gateway Protocol (BGP), equal-cost multipath (ECMP) hashing and leaf-spine topologies. AI traffic stresses them as web traffic never does, with a few enormous, synchronized, low-entropy flows instead of millions of small ones. This chapter…

• AI backend traffic is few, huge, synchronized, low-entropy flows: design for the worst path.
• Route to the host: a /31 per NIC, every link active, regular addresses that make validation automatic.
• Keep MTU consistent per link class (9216 / 9000 / 4096 on ORION); mismatches blackhole large frames while BGP stays up, and one probe tests one ECMP path.
• RFC 7938 eBGP: shared spine ASN, unique leaf ASNs, inbound filters, and maximum-paths covering every equal-cost path (≥ 32 at an ORION leaf).
• ECMP balances flows, not bytes: eight flows on eight links collide 99.8 % of the time.
• One link down costs 3 % of raw uplink but 12.5 % of ECMP-aware ingress; a spine costs 25 %; a rail leaf removes one NIC from every node.
• In an ECMP fabric, “intermittent” usually means “one path in N”: vary the 5tuple before trusting a probe.

Sections: What AI traffic does to a data-center network · Routed to the host: Layer 2 and addressing · MTU, path MTU discovery and blackholes · Routing the fabric: eBGP in the data center · ECMP: how traffic is actually spread · Leaf-spine arithmetic and the rail-optimized topology · Failure behaviour and convergence

A 70-billion-parameter training step on ORION moves tens of gigabytes between GPUs every few seconds, and the step cannot finish until the slowest transfer does. TCP sockets spend CPU cycles, memory bandwidth and microseconds on every byte; at 400 Gb/s per network interface card (NIC) and eight NICs per node, that tax is unaffordable. Remote Direct Memory Access (RDMA) removes the kernel and the CPU from the data…

• Kernel bypass puts the wire, PCIe and memory in charge of performance; the QP’s RTR/RTS settings (GID index, path MTU, ACK timeout) are where RDMA is usually misconfigured.
• InfiniBand is lossless by credit-based flow control under a subnet manager.
• RoCEv2 carries the IBA transport in UDP port 4791; DSCP selects the lossless class, the UDP source port gives ECMP entropy, and the GID index selects RoCEv1 versus v2.
• Go-back-N makes a mid-message loss cost about one RTT of resent data and a tail loss a full ACK timeout: RoCE runs lossless or relies on selective repeat.
• A 4,096-byte path MTU gives 98.0 % efficiency (≈ 392 Gb/s at 400 Gb/s); 1,024 gives 92.6 % at 3.8× the packet rate. MTU must agree at netdev, RDMA and switch layers.

Sections: Why RDMA: the cost of moving a byte through a kernel · The verbs model · RoCEv2: the IBA transport over UDP/IP · InfiniBand: a fabric designed for RDMA · Transport behaviour: what happens when a packet goes missing · Path MTU and wire efficiency · Multi-rail design and the RoCEv2–InfiniBand choice · Tools and the evidence they give

An AI training job is only as fast as its slowest collective, and a collective is only as fast as its most congested queue. Remote direct memory access (RDMA) over Converged Ethernet version 2 (RoCEv2) fabrics make Ethernet “lossless” by combining priority flow control (PFC), which stops a sender before a buffer overflows, with Explicit Congestion Notification (ECN) and Data Center Quantized Congestion Notification…

• Congestion lives in egress queues. Buffers are microseconds deep at 400 Gb/s, so PFC (or loss) handles an incast’s first RTT and DCQCN everything after.
• ECN thresholds anchor on the BDP (≈ 200 KB same-rail, 250 KB cross-rail); Kmin must mark before PFC fires.
• Headroom = 2 × MTU + 2 × r × L/v + r × t_resp + PFC frame: ≈ 93–119 KB per port per lossless priority on ORION, ≈ 6.8 MB per leaf (static worst case), and the same again for each extra lossless priority.
• Because PFC stops a whole (port, priority), one hot flow can stall others: HOL blocking, congestion trees, storms and deadlock.
• A healthy lossless fabric has far more ECN marks than pause frames and zero lossless-class discards; incast with only sporadic marks means Kmin sits above Xoff.

Sections: Why AI fabrics care about congestion · Queues, buffers and where congestion lives · ECN: marking before it hurts · PFC mechanics and headroom · The risks of PFC · DCQCN: end-to-end congestion control · Incast arithmetic · Congestion telemetry: reading the signatures

A 256-GPU training job is only as fast as its worst link, and in an AI fabric the worst link is usually not congested. It is dirty, bent, mis-seated, mis-patched, or the wrong part. ORION’s backend has 512 links and 1,024 transceivers, each a 4-lane four-level pulse-amplitude modulation (PAM4) system that runs by design at a raw bit error ratio (BER) that needs forward error correction (FEC) to look perfect. This…

• Healthy PAM4 links have non-zero raw BER; uncorrectable codewords are never normal.
• FEC is a cliff: track margin in decades and its trend; bursts show in the histogram.
• Cut-through stomps corrupted frames, so CRC propagates downstream and uncorrectables do not.
• Excess loss and lane imbalance warn before DOM alarms.
• Validate cabling with LLDP and commission every link through explicit gates.

Sections: Why the physical layer dominates AI-fabric operations · Inside the switch ASIC · The NIC↔switch path, PAM4, and FEC · Transceivers and media · Link budgets and DOM · Error counters: physical or higher-layer? · Cleaning, cabling, commissioning, and spares

A 70-billion-parameter model needs about 1.1 TB of training state and roughly 1.8 × 10¹⁸ floating-point operations per optimizer step. No single GPU holds the first number or finishes the second in useful time, so the work is cut up across hundreds of GPUs, and every cut creates communication. Data, tensor, pipeline, context, and expert parallelism are five ways of cutting, each with its own traffic pattern and…

• A step costs ≈ 6ND FLOP; MFU measures how much peak becomes useful work.
• Memory = 16 B/param model state (sharded by TP, PP, ZeRO) + ≈ s·b·h·34/t activations per layer per in-flight micro-batch + overhead.
• DP moves ≈ 2× the gradient bytes once per step; ZeRO-3 costs 1.5× per microbatch.
• TP volume ∝ (t − 1), 1.2 TB per step here, and belongs inside the NVLink domain.
• The bubble does not shrink as GPUs grow, capping strong scaling (72 % at 16× in the model). τ_opt = √(2·C·MTBF): ≈ 66 min on ORION-256, ≈ 16 min and 17.5 % waste on ORION-XL.

Sections: Anatomy of a training step · Memory accounting per GPU · Data parallelism · Tensor parallelism · Pipeline parallelism · Context and expert parallelism · Topology-aware placement · Scaling efficiency · Failure modes and a failure-aware workflow

Chapter 11 turned every parallelism strategy into a list of collectives: an all-reduce of gradients, a reduce-scatter and all-gather per layer, an all-to-all per mixture-ofexperts (MoE) layer. This chapter opens those boxes. A collective’s cost depends on message size, rank count, algorithm, protocol and topology. The NVIDIA Collective Communications Library (NCCL) chooses the algorithm and protocol at run time from…

• A ring moves 2(n − 1)/n × S per rank; trees cut latency to O(log n); in-network reduction halves bytes per GPU. n½ = α·B/factor tells you whether collectives are latency- or bandwidthbound.
• Compare busbw against link limits; ignore “Avg bus bandwidth”.
• NCCL’s INFO log and topology dump are the NCCL_ALGO/NCCL_PROTO are for testing hypotheses only. primary evidence; Steady low plateau → mapping or link speed; low and jittery → errors, retransmission, or congestion; #wrong > 0 → correctness incident.

Sections: From parallelism to collectives · The collectives · Cost models: alpha-beta for ring, tree, and friends · Reading nccl-tests: algbw, busbw, and the traps · Inside NCCL: topology, channels, algorithms, protocols · Rank mapping, rail alignment, and failure signatures

Chapter 11, Distributed Training Architecture, and Chapter 12, Collective Communication and NCCL, counted bytes and timed collectives. This chapter follows the bytes. A gradient chunk leaving GPU 3 on one ORION node crosses highbandwidth memory (HBM), a PCI Express (PCIe) switch, a network interface card (NIC), a rail leaf, possibly a spine, another NIC and another PCIe switch, and lands in another GPU’s HBM, with…

• An inter-node byte crosses PCIe twice, a NIC twice and one leaf (rail-local) or leaf–spine–leaf; NVLink carries ≈ 122× more bytes per step than the NIC.
• Host staging needs two DRAM copies and four PCIe transfers per message (GPUDirect RDMA: zero and two), turning W-TRAIN’s DP reduction from NIC-bound (0.17 s) into PCIe-uplink-bound (≥ 0.56 s).
• Keep GPU i on NIC i and ranks on their NUMA node; NODE/SYS pairs lose GPUDirect RDMA, while PXN reaches another rail through one NVLink hop and a PIX path.
• Lost GPU-time = Σ(max − own); at p = 0.1 % per rank, 98 % of steps at 4,096 ranks contain a slow rank.
• GPUDirect RDMA is proven by /GDRDMA lines, CUDA-buffer perftest at line rate and a GDR-on/off comparison, never by installed packages.

Sections: What actually moves in a training step · GPU-to-GPU inside a node: NVLink and peer-to-peer · GPU-to-NIC: PCIe, locality and GPUDirect RDMA · The end-to-end path and its copies · Locality and overlap · Stragglers and their amplification · “GPU utilization low while the network is busy” · Validating GPUDirect RDMA and locality

A 256-GPU training job reads its dataset continuously, writes a checkpoint of nearly a terabyte every few minutes, and, when something fails, must find, validate and load the last good checkpoint before 256 accelerators can work again. Whether the input pipeline is metadata- or bandwidth-bound decides the file format and worker count. Whether checkpoints are synchronous or asynchronous, local or remote, decides how…

• NVMe follows Little’s law; with 256 ranks 92 % of steps see someone’s p99, so accept storage on p99/p99.9.
• A full-cluster ORION checkpoint is bound by the storage servers’ 80 GB/s, not NICs or fabric.
• Small-file pipelines are metadata-bound (0.2 GB/s per node in the lab), sharded ones bandwidth-bound (7.6 GB/s): fix with a format change, not more workers.
• W-TRAIN’s 980 GB checkpoint blocks ≈ 17.8 s synchronously and ≈ 0.65 s asynchronously, at the price of staging memory, interference and completion latency.
• A checkpoint exists only when committed; validate before restore, never trust latest, and retain validated checkpoints.
• Pick τ ≈ √(2δM) rounded to steps; compare strategies in lost GPU-hours/day (55.6 async versus 121.1 sync at M = 48 h), then attack time-to-resume.

Sections: Why storage decides cluster goodput · NVMe and the host↔storage path · Parallel file systems and object storage · Metadata-bound versus bandwidth-bound pipelines · Tail latency and measuring storage · Checkpoint anatomy and asynchronous saves · Atomic commit and restore validation · Recovery economics and the optimal interval

Serving runs millions of short, unpredictable jobs per day, each with a user waiting. A large language model (LLM) request has two phases with opposite hardware appetites: prefill processes the whole prompt in one compute-bound pass, and decode produces one token at a time, bound by memory bandwidth. Between them sits the key-value (KV) cache, a per-request allocation that grows with every token and usually decides…

• Prefill is compute-bound (≈ 91 ms for a 2k prompt on HX8); decode streams all weights plus all running KV each step.
• W-INFER’s KV is 320 KiB per token; usable concurrency is well below theory (about 140 versus 186 4k sequences on TP = 4 FP8).
• Continuous batching and paged KV enable large batches but bring prefill interference and preemption, visible in ITL first.
• Size pools with Little’s law and M/D/1 percentiles, not means; queues outlive bursts, and for an 8:1 mix prefill sets the pool size.
• Disaggregation adds a 671 MB-per-request transfer (214.7 Gb/s at 40 req/s): nearly free over rail-local RDMA, infeasible over TCP at 6:2.
• State every benchmark’s configuration; in incidents, decompose TTFT and group by replica and tenant.

Sections: Anatomy of an inference request · Prefill and decode: two different machines · The KV cache and the memory budget · Batching, paged KV and preemption · Queueing, Little’s law and phase SLOs · Choosing a replica layout · Disaggregated prefill and decode · Speculative decoding

A GPU cluster is not built once; it is built every day. Nodes arrive, get imaged, fail acceptance, get repaired, get upgraded in waves, fall off the bus at 3 a.m., and are eventually wiped and retired. Every transition is a chance to put a subtly broken machine into a 256-GPU job, where one throttling GPU silently taxes every other GPU in the job. This chapter treats the fleet as a system with a lifecycle, from…

• One degraded node taxes a whole synchronous job: one GPU throttled to 75 % costs ≈ 54 GPUs of output on ORION [MODELED], so fleet-wide cost justifies acceptance depth.
• Guarded transitions, a single writer and an append-only audit log are the safety mechanism: nodes enter production only after acceptance.
• Acceptance runs cheap checks first, proves the GPUDirect RDMA path and link quality, not only bandwidth, and ends with burn-in and an algorithm-aware NCCL test judged on medians.
• Compatibility is validated bundles plus a blocked list, checked per device; repaired hardware returns only through re-acceptance.
• Staged upgrades need a canary in separate racks, bake sized by λNT, per-wave and per-rack caps, and pre-declared abort conditions.

Sections: The fleet as a system · Inventory, out-of-band control and provisioning · Node acceptance testing · Firmware and driver compatibility matrices · Staged upgrades: canary, waves and rollback · Repair, spares and decommissioning

A GPU cluster is only as productive as the decisions that put work onto it. ORION has 256 GPUs, but at any moment some are allocated, some sit on nodes draining for a firmware wave, some are carved into MIG (Multi-Instance GPU) slices, some are reserved for one tenant, and the free ones may be scattered in a way no 32-GPU training job can use. This chapter explains how Kubernetes and Slurm decide who runs and where,…

• A GPU is schedulable only if it survives every filter and sits in a usable shape; “free GPUs” on a dashboard is not that number.
• Kubernetes trusts allocatable; a failed device plugin or MIG mixed makes it diverge from nvidia-smi. Labels attract, taints repel, and dedicated nodes need both.
• Training needs gangs (partial allocation can waste 256 GPU-hours overnight [MODELED]); quota is policy, not capacity; preempting training costs ≈ 32 GPUhours per 64-GPU job [MODELED], so make preemption asymmetric.
• On ORION’s flat fabric, topology means node-then-socket best fit, whole nodes for TP = 8, rail completeness for gangs and rack-compact placement.
• A Kubernetes/Slurm hybrid is safe when the Chapter 16 lifecycle system is the single source of node state.

Sections: The scheduling problem on a GPU cluster · How a GPU becomes schedulable in Kubernetes · Sharing a GPU: MIG and time-slicing · Gangs, quotas, priority and fair share · Slurm for GPU training · Topology-aware placement on ORION · Kubernetes or Slurm?

A 256-GPU training job that runs 18 % slower for two hours does not crash, does not log an error, and does not trip a single classic infrastructure alert, yet it burns more accelerator time than most hard outages. AI clusters fail quietly: one hot link, one throttled GPU or one bad switch profile stretches every synchronous step for every rank. This chapter builds the observability system that catches those…

• Workload progress detects; component metrics explain.
• DCGM_FI_DEV_GPU_UTIL stays ≈ 97 % while NCCL waits; SM activity, tensor activity and power show whether GPUs do math.
• Watch pauses and ECN marks, not discards, on the lossless class; stream 64bit switch counters.
• Page on multiwindow burn; a training burn alert would have caught INC-2291 in ≈ 15–17 minutes, not 48.
• In a synchronous job the straggler’s activity rises against its peers; report TTD, TTM and TTR separately.

Sections: Why AI clusters defeat conventional monitoring · The telemetry stack · Signals that tell the truth · Prometheus for GPU fleets · SLIs and SLOs for AI infrastructure · Alerts: page on symptoms, using burn rates · Dashboards, incident command and postmortems

A 4,096-GPU training job built from components that each fail once every couple of months will still be interrupted every few hours. At that scale, failure is not an incident class; it is the operating environment. This chapter teaches how to walk from a hung rank to the exact GPU, host channel adapter (HCA) and switch port in minutes, quarantine the smallest defensible failure domain, and restart once from a…

• Job MTBF = node MTBF / nodes: 48 h for 32 nodes, 3 h for 512; restart overhead dominates lost goodput.
• FMEA is ordinal: act on high severity regardless of RPN.
• An NCCL timeout localizes nothing; ranks behind and NET/IB errors at onset do, and the retry budget models which flaps survive.
• Silent data corruption raises no error: catch it with loss/gradient-norm anomalies, cross-replica checksums and golden-batch replay, and restore from before the onset.
• Quarantine the smallest defensible domain, preserve evidence before resets, and never auto-restart into a repeat offender; a checkpoint exists only once its metadata is committed.
• Tenant isolation spans identity, secrets, network, storage, GPU allocation and residue, device access, firmware, supply chain and decommissioning; a threat model maps each asset to a read-only check.

Sections: Failure is the steady state · Fault domains, FMEA, and blast radius · GPU and host error taxonomy · Distributed failure signatures · The fast isolation workflow · Containment and recovery · Security and multi-tenant isolation

A GPU cluster is changed far more often than it is built: firmware bundles, quality-ofservice (QoS) profiles, kernel patches, maximum transmission unit (MTU) corrections, scheduler policy. At fleet scale those changes are made by automation, which is also the fastest way to take the whole cluster down: one wrong value in one template, applied in parallel, breaks all 256 of ORION’s GPUs in under a minute. This…

• Changes, not hardware, cause the largest outages, because automation copies one defect everywhere.
• Build level-triggered reconcilers whose second run is a provable no-op; give every field one writer, and treat a timeout as unknown.
• Cross-object policy-as-code encodes what schemas cannot: host_mtu + 22 ≤ port_mtu, rdma_mtu + 64 ≤ host_mtu, PFC on exactly the RoCE priority.
• Stratify canaries by every attribute the change touches, because detection = 1 − (1 − p)^k is low for a minority defect.
• Abort conditions are pre-declared statistics sized against noise, with absolute floors, zero tolerance for hard faults, and hold on missing evidence.
• Price a rollout by the work it stalls: with no spares, every training drain stops the gang.
• Rollback needs per-target before-state, a rehearsed procedure and a predecided scope; dry run, caps, kill switches and audit records make automation production-safe.

Sections: Why automation is the largest blast radius in the building · Idempotency, desired state, and reconciliation · Validation: catching the defect before any target is touched · CI/CD and GitOps · Staged rollout: canaries, waves, bake time, and abort conditions · Rollback and fleet-safety controls

“We have 1,024 GPUs” is the most expensive sentence in AI infrastructure planning. Some of those GPUs are not energized, some are in repair or held as spares, some sit in fragments too small for the jobs in the queue, and the ones that run spend part of their time waiting on the network or redoing lost work. This chapter turns nominal GPU counts into usable and useful capacity, forecasts when that capacity runs out…

• Report utilization as a ladder (usable → allocated → goodput → MFU); dashboards usually show the active rung.
• Usable = min(installed, energizable) × (1 − maintenance) × (1 − repair) × (1 − fragmentation); ORION’s factor is 0.850.
• Plan to required = demand / ≤ 85 % allocation; facility power has the longest lead time (≥ 4 quarters), and a shortfall can be power-caused while GPUs sit stranded.
• ORION-256 costs ≈ $6.73M a year: $3.00 per nominal, $4.16 per allocated, $4.72 per useful GPU-hour; electricity is ≈ 4 %.
• Compare designs on useful throughput per dollar and per watt at the binding constraint.
• Redundancy, derating and transient margin shrink plannable rack power; the lightest cap that fits one more node can add 48 % throughput.

Sections: What “utilization” means: four definitions · Nominal versus usable capacity · Demand forecasting and onboarding lead times · Total cost of ownership · Throughput per dollar and per watt: comparing designs · The facility envelope: power, cooling, and transients · Power capping and workload/facility coordination

A distributed training job is one synchronous machine built from GPUs, hosts, PCI Express (PCIe) switches, network interface cards (NICs), a backend fabric, storage, a scheduler and software. When its step time regresses, the cause can sit in any layer, and production rarely offers one clean change to blame: driver upgrades, placement policies, storage moves and container rebuilds land in the same window while…

• In a lockstep job the slowest part paces everyone, and symptoms surface far from their cause.
• Rank tests by information value ÷ (cost × risk); the best first test partitions data you already have.
• Contribution = effect × change in exposure; fit all factors together to avoid confounding.
• A named cause needs statistical, direct and model evidence in agreement.
• Know your MDE, tune one knob at a time against a prediction, and report facts, causes and uncertainty apart.

Sections: Why cross-layer regressions are hard · Evidence by layer · From symptom to a ranked hypothesis tree · Natural experiments and attribution · Scaling-efficiency analysis · Benchmark design and evidence-based tuning · Anti-patterns and the before/after report

Real work arrives as “design the next cluster”, “prove this one is ready”, or “tell us why it is slower than the plan said”. This chapter is a guide to running that kind of project end to end, from framing and decisions through arithmetic, failure domains, honest evidence and a safe failure exercise to a design review and a backlog. A worked mini-capstone on the ORION reference cluster ties Chapters 1–22 together.

• A capstone is a reviewable chain of reasoning; every load-bearing number has a label, owner and validation.
• Rank assumptions by decision value: MFU 0.40 → 0.35 turns a 106.7-day plan into a 121.9-day one that misses its objective.
• Record decisions as ADRs with revisit triggers; run consistency arithmetic, with rack power at maximum draw, before debate.
• In a rail-optimized fabric, spines must be justified, and a rail leaf has a clusterwide logical blast radius.
• With δ = 60 s and M = 48 h, failures waste ≈ 3 % of goodput; asynchronous saving beats doubling PFS bandwidth.
• Correlated failures come from change batches: size rollouts to one DP replica.
• Label every benchmark row; in a failure exercise, validate resumption, not restart.

Sections: What a capstone is for · Framing the problem and the assumptions register · Architecture as a set of decisions · Consistency arithmetic · Failure-domain analysis · Design review, benchmark evidence and the failure exercise · Presentation and the next-step backlog

This chapter turns the book’s knowledge into a career you can prove. AIinfrastructure roles are new, titles are inconsistent, and interviews probe across layers, so a network engineer is asked about NVIDIA Collective Communications Library (NCCL) rings and a platform engineer about remote direct memory access (RDMA) over Converged Ethernet (RoCE) congestion. We map six roles to competencies, define…

• Every role is tested on the diagnostic loop.
• Evidence type caps proficiency, and any two-level gap blocks “ready”.
• Levels differ by scope, ambiguity, judgment, operational ownership and influence, not years.
• Job-market claims need a dated register deduplicated by role, not only by posting.
• Design interviews reward labelled assumptions and derived numbers.
• Label every portfolio number and never present simulation as measurement.

Sections: What the job actually is · Competency maps for six roles · Levels: junior to principal by observable behaviour · The employer landscape, and how not to fool yourself about it · System-design interviews · Case studies, portfolio and engineering practice

ORION is the book’s running example: a 256-GPU training and inference cluster operated by Halcyon AI, a fictional company. ORION is an illustrative, H100class teaching reference, not a measured system. Nobody built it and nothing in this appendix was measured on it. Every value is [ILLUSTRATIVE] unless it carries another label. Values that approximate a public vendor specification are marked [VENDOR-SPEC, 2026-10]…

• ORION is the book’s running example: a 256-GPU training and inference cluster operated by Halcyon AI, a fictional company.
• ORION is an illustrative, H100class teaching reference, not a measured system.
• Nobody built it and nothing in this appendix was measured on it.
• Every value is [ILLUSTRATIVE] unless it carries another label.

Sections: A.1 The HX8 node · A.2 The ORION-256 cluster · A.3 Reference workloads · A.4 The ORION-XL variant · A.5 Reading the evidence labels

A performance number without its conditions is a rumour. This appendix collects, in one place, the benchmark discipline that Chapter 6 (GPU Performance Engineering), Chapter 12 (Collective Communication and NCCL), Chapter 15 (LLM Inference and Serving Infrastructure) and Chapter 22 (Cross-Layer Performance Engineering) apply to their own measurements, and that the companion repository’s lab standards require of…

• A performance number without its conditions is a rumour.
• This appendix collects, in one place, the benchmark discipline that Chapter 6 (GPU Performance Engineering), Chapter 12 (Collective Communication and NCCL), Chapter 15 (LLM Inference and Serving Infrastructure) and Chapter 22 (Cross-Layer Performance Engineering) apply to their own measurements, and that the companion repository’s lab standards require of every L2 and L3 run.
• Use it as the minimum for any performance claim you publish, put in a design review, or use to justify a change.
• The standard has three parts: a core report that every claim needs, extra fields for distributed training and for inference, and a template to copy.

Sections: B.1 The minimum report · B.2 Extra fields for distributed training · B.3 Extra fields for inference · B.4 Report template

Use this checklist before a cluster, a new block of nodes or a major upgrade goes into service, and again after any change that touches several layers. Each row is a concrete check, the criterion that counts as a pass, and (under “Ch.”) the chapters that teach the mechanism behind it. The pass criteria are ORION’s and are [ILLUSTRATIVE] unless marked otherwise: derive your own thresholds from known-good nodes,…

• Use this checklist before a cluster, a new block of nodes or a major upgrade goes into service, and again after any change that touches several layers.
• Each row is a concrete check, the criterion that counts as a pass, and (under “Ch.”) the chapters that teach the mechanism behind it.
• The pass criteria are ORION’s and are [ILLUSTRATIVE] unless marked otherwise: derive your own thresholds from known-good nodes, typically the median minus a margin set by run-to-run spread (Chapter 16).
• A check that cannot be run is recorded as “not run”, never as a pass.

Sections: C.1 Facility · C.2 Host · C.3 GPU · C.4 Network and RDMA · C.5 Storage and checkpointing · C.6 Scheduling · C.7 Observability · C.8 Reliability and security · C.9 Change management · C.10 Cost and capacity

Terms and acronyms used in the chapters, defined as the chapters use them. Numbers in the definitions are ORION’s reference values and carry their evidence labels; Appendix A has the full reference configuration. The chapter that introduces a term gives its mechanism, its numbers and its caveats. 1F1B — One-forward-one-backward: a pipeline schedule that alternates forward and backward micro-batches, so stage 0 holds…

• Terms and acronyms used in the chapters, defined as the chapters use them.
• Numbers in the definitions are ORION’s reference values and carry their evidence labels; Appendix A has the full reference configuration.
• The chapter that introduces a term gives its mechanism, its numbers and its caveats. 1F1B — One-forward-one-backward: a pipeline schedule that alternates forward and backward micro-batches, so stage 0 holds activations for at most p micro-batches. 2N, N+1 — Redundancy levels: 2N duplicates the whole path (A and B feeds, each able to carry the load); N+1 adds one spare module to N.
• A/A comparison — Running the same code in both arms of a benchmark to measure the noise floor.

Sections: Glossary

Claim Ch. Applies to Source Dense BF16 ≈ 989 TFLOP/s, dense FP8 ≈ 1,979 TFLOP/s (the datasheet’s 1,979 / 3,958 are with sparsity) 4, 6, 11, H100 SXM NVIDIA H100 Tensor Core GPU 2026-10-30 (product page and datasheet) [1]

• Applies to Source Dense BF16 ≈ 989 TFLOP/s, dense FP8 ≈ 1,979 TFLOP/s (the datasheet’s 1,979 / 3,958 are with sparsity) 4, 6, 11, H100 SXM NVIDIA H100 Tensor Core GPU 2026-10-30 (product page and datasheet) [1]

Sections: E.1 GPU hardware · E.2 NVIDIA driver, nvidia-smi, CUDA and GPUDirect · E.3 DCGM and dcgm-exporter · E.4 NCCL and nccl-tests · E.5 PyTorch and Megatron-LM · E.6 vLLM · E.7 Kubernetes and the NVIDIA GPU Operator · E.8 Kueue · E.9 Slurm · E.10 Host, PCIe and RDMA · E.11 Ethernet, routing and optics · E.12 Storage · E.13 Profiling, GPU sharing, container security and routing · E.14 Unverified claims · Further Reading · About the Author