AI-Driven Chaos Engineering:
Breaking Cloud Infrastructure and Applications on Purpose, with Prompts, Guardrails, and Evidence

• Most outages come from ordinary things happening at the wrong moment; chaos engineering makes those moments happen on purpose, in a controlled way, so you find the weakness before customers do.
• AI for chaos: an assistant drafts experiments quickly, so review becomes the bottleneck. In this book AI designs, validates and analyzes experiments but never executes them; people start runs and decide what results mean.
• Chaos for AI: GPU training, inference, retrieval, agents and guardrails fail differently from a web shop; failure in large training is a rate, not an event.
• Coverage beyond the web shop: AI infrastructure (Part V, Chapters 18–20), security chaos (Chapter 15), production practice (Part VI) and three clouds (AWS and Local in full; Azure and Google Cloud as binding notes; dated specifics in Appendix B).
• Every lab is driven by prompts and has a review checklist of the mistakes AI assistants actually make; a Safety Contract, a validator, opt-in flags, run permissions and account guardrails keep experiments safe.
• The book has 76 labs, 3 exercises and 3 capstone projects, indexed in catalog/labs.yaml in the companion repository github.com/sbittla/chaos-engineering. The Local route and CPU profile cover the Core path: no GPU or cloud account needed to start.
• The case study is Kestrel: Kestrel Commerce (a microservice shop) and Kestrel Assist (an AI support assistant), with ten postmortems as shared evidence.
• For SREs, platform, performance, backend, ML infrastructure and security engineers, engineers adopting AI tooling for operations, and staff/principal engineers who decide where a chaos program spends its effort.

Sections: Two halves: AI for chaos, and chaos for AI · What this book covers · Who this book is for · What makes this book different · How to Use This Book

This first chapter builds on your own experience of incidents. It sets up two ideas the rest of the book depends on: the two architectures (traditional cloud and AI infrastructure) and the case study, Kestrel, with its incident history. Chapter 2 turns both into a working lab and a no-op experiment that proves the abort path.

• Distributed systems fail by interaction: in INC-2041 every component worked as built, and together they produced 7 minutes of errors.
• Chaos engineering is experimentation, and a fix counts only when a re-run verifies it. This book starts in preprod, earns production, and runs experiments with load.
• AI moves the bottleneck from writing to reviewing. The assistant drafts; people approve, a pipeline executes, and deterministic checks back every review.
• Traditional and AI infrastructure differ on eight dimensions, starting with the unit of failure.
• At large training scale failure is a rate, and slow failures cost as much as dead ones.
• Latent failures are always present, outages need several causes, and people are part of the system, so measure detection and adaptation, not only survival.
• Kestrel’s ten postmortems are the book’s shared evidence, each with an unverified fix to test.

Sections: Distributed systems fail by interaction · What AI changes: from writing experiments to reviewing them · Two architectures · Complex systems and resilience engineering · Meet Kestrel: the case study and its incident history

Chapter 1 gave you the scientific loop, the two architectures (traditional cloud and AI infrastructure), and Kestrel with its incident history. This chapter turns them into machinery you can run: the Kestrel Commerce cluster on your laptop, the Kestrel Assist model stack in Docker, the contracts and validator every lab uses, and a golden path that proves all of it from a clean start. It ends with three labs that…

• Kestrel has two systems: Commerce (web-bff, checkout, inventory, Postgres, Redis, and optional Kafka on AWS) and Assist (RAG, an agent, a guardrail, inference, training). Local and AWS are printed in full; Azure and Google Cloud are binding notes.
• On the local route, the stop path is Prometheus alerts labeled chaos_abort="true" , routed by Alertmanager to local-abort , which deletes Chaos Mesh objects. The kill switch is a second path that needs only kubectl .
• Only AWS documents an automatic stop tied to monitoring. On Azure you build the alertto-cancel chain; on Google Cloud the stop comes from the in-cluster tool (Section 4.4).
• The AWS route costs roughly $2–4 per hour [verify] plus $0.10 per FIS action-minute; the local route costs memory and disk. Tear down every session; do Plan, Generate, and Verify with nothing running.
• SC-1 bounds the assistant, CP-1 is its only source of facts, and OC-1 is the strict shape of every plan. The validator fails closed, including when it checked nothing; 39 offline negative tests (43 with --live ) prove it.
• The prompt chain is Plan, review, Generate, Verify, Execute, Analyze. The AI never executes.
• The AI stack serves open-weight models behind LiteLLM aliases, with Qdrant and a fault proxy. It picks gpu only if Docker can see the GPU; on CPU the numbers change and the failure behavior doesn’t.
• A fallback can hide a failure: check which model answered, not only that a request returned 200.

Sections: The reference environments · What it costs, and how to keep it low · Before you begin: tools and tested versions · The starter kit · The prompt chain and how to run it · The local AI lab stack: open-weight LLMs, embeddings, and a vector database in Docker

Chapter 2 gave you a working local platform, the AI lab stack, and the golden path, and Lab 2.1 showed that an abort alert makes local-abort delete a running Chaos Mesh object. That proof assumes the fault is something the abort path can see. This chapter shows what a fault actually changes in the kernel, the runtime, the proxy, the cluster, or the cloud API, and why the mechanism decides both the blast radius and…

• netem delays egress packets, not requests; loss shows up as stalls. DROP tests timeouts, REJECT errors.
• L7 faults can return status codes and be scoped to headers, if every hop propagates the header. Pod deletes can’t be scoped to synthetic traffic.
• FIS EKS pod actions that inject inside the pod use ephemeral containers (hence their RBAC and security-context requirements); Chaos Mesh uses a privileged per-node daemon and a finalizer that reverts the fault before the object goes away.
• Where a rule lives sets its blast radius: a node-level tc rule hits every pod on the node and the node’s own traffic.
• An abort path removes only what it knows about. A hand-applied fault survives every alert and the kill switch; prove cleanup from the effect, and make the kill switch carry on past a failed stop (Section 6.7).
• Choose tools on scope, stop conditions, RBAC and audit, opt-in, multi-cloud reach, cost and support, and project health. The book uses AWS FIS plus Chaos Mesh because one in-cluster engine runs on every route and FIS has the only managed, alarm-bound stop.

Sections: What a fault actually does · The chaos tool landscape, and how to choose

Chapter 3 showed what a fault actually changes and how each tool reverts it, and Lab 3.1 left you with a hard lesson: a fault the stop path can’t see is never stopped. It also chose AWS FIS and Chaos Mesh. This chapter supplies the other half of stopping: a definition of steady state that says when things are no longer fine, thresholds that turn that definition into decisions, stop paths that act on them, and proof…

• SLIs are good ÷ total; SLOs set targets over a window; the error budget (43.2 minutes, or about 1.04 million requests a month, for checkout) is what experiments spend.
• AI outputs need TTFT, grounded-ratio, guardrail-coverage, and sampled quality SLIs; INC-2267 and INC-2290 both kept error rates at zero.
• Burn rate = error ratio ÷ (1 − SLO); 14.4, 6, and 1 spend 2%, 5%, and 10% of the budget in 1 h, 6 h, and 3 days. A 1% error ratio never trips the 14.4× page, so burn-rate alerts are a backstop. Check predictions with promtool test rules written to pass on the Prometheus version you run.
• Acceptance judges the hypothesis, abort protects users now, the SLO backstop protects the month.
• Check the abort against the experiment budget: below the threshold, exposure is threshold × duration, so cap the fault’s duration (about 650 s for a 13 error-second budget at 2%) or add a budget-consumed abort.
• Stop paths: FIS stop conditions on CloudWatch alarms; Alertmanager → local-abort locally; alert → automation → cancel or delete on Azure and Google Cloud. Measure abort delay from fault start, three times.
• Use absent_over_time(...[1m]) , not absent() ; break each pipeline stage while a harmless fault runs; close the blind spots with an outside watchdog whose detection time counts the hold after Prometheus dies.
• Enforce observability for chaos as a readiness gate.

Sections: SLIs, SLOs, and error budgets, for deterministic services and non-deterministic AI outputs · Burn rates and how the standard thresholds are derived · Three thresholds, three jobs: acceptance, abort, SLO backstop · Stop paths · Monitors are part of the system under test · Observability for chaos

Chapter 4 gave you steady state you can measure, SLOs, stop paths whose delay you’ve timed, and monitors that abort when their own data goes missing. None of that means much without traffic: a steady state measured on an idle system and an abort alarm that never sees a request prove nothing. This chapter supplies the traffic. You build a workload model for Kestrel Commerce, learn why the shape of load matters as…

• Fault impact depends on load: latency rises steeply near the utilization limit, and Little’s law says how many requests a fault can catch. Run under at least typical peak, scaled per pod.
• Build workload models from load balancer logs, aggregated inside production; never export personal data.
• Use open models for user traffic; closed models hide overload and omit the slowest samples (coordinated omission). Size maxVUs by Little’s law from worst-case latency.
• Match the profile and arrival process to the fault; even arrivals understate queueing.
• Model inference load in tokens: prompt and output distributions, streaming, TTFT, ITL, and goodput.
• Match per-pod load and the percentage of capacity lost, not counts. The local route tests mechanisms only.
• Keep generators outside the system under test, monitored, and spreading requests across pods.
• Apply the validity gate before any verdict; it fails closed and says what it didn’t measure.
• Start load before faults, keep phases on one timeline and one set of stop conditions, and check the generator delivered the model. A 100% blast radius is a local-only exception; shared environments stay inside the cap.

Sections: Why chaos needs load · Building a workload model from production · Open and closed load models; coordinated omission · Load profiles · Token-aware load for AI inference · What scales down, and what doesn’t · Load generator hygiene and the validity gate · Load tools and services · Composing load and faults · Validating the model

Chapter 5 gave you load you can trust and a validity gate; Chapter 4 gave you the automatic stop path and the monitors behind it. Before Chapter 7 puts a real fault into Kestrel Commerce under that load, this chapter decides where a fault may land and who may start one. It is the one home for guardrails: every later “opt-in flags (Chapter 6)” or “run controls: Chapter 6” means the controls built here. It hands…

• Safety is five independent layers. Layers 2 to 5 hold even if the AI is compromised; prove each with the others off.
• The abort (automatic, one experiment), the freeze (manual, new starts), and the kill switch (manual, every running fault) are three controls with three jobs.
• Chaos is opt-in: without a valid six-field flag, a resource is not a target, and unknown means unsafe.
• Enforce flags on AWS with IAM conditions and a tag-protecting SCP; on Kubernetes with labels, the chaos tool’s filter, and admission that checks every selected namespace; and in the pipeline, against live labels.
• The freeze blocks every start path and never blocks a stop.
• Stop is broad, start is narrow, and flagging is separate from both; every run records who proposed, approved, and ran it, and what ran.
• Pipelines start only from the protected branch, through a reviewed environment that prevents self-review, with OIDC trust bound to one repository and environment and actions pinned to SHAs.
• The kill switch is manual, covers every scope, survives one failed stop, and proves the effect is gone.

Sections: Layers of safety: prompt, validator, approval, account, kill switch · Default deny: chaos is opt-in · The chaos-enablement flag · The global freeze flag · Authorization matrix: who may propose, approve, start, and stop · Run controls in the pipeline · The kill switch

Chapters 2 to 6 built everything an experiment needs before it touches anything: the golden path (Chapter 2), fault mechanics and cleanup (Chapter 3), SLOs and proved abort paths (Chapter 4), a validated workload model and the validity gate (Chapter 5), and, in Chapter 6, the opt-in flags, run permissions, and kill switch. This chapter spends all of that on Kestrel’s first real experiment, then steps back to decide…

• Pod deletion is a set of parallel events; errors come from the gaps between them, and latency from a replacement that’s Ready before it’s warm. A pod delete can’t be confined to synthetic traffic.
• The first experiment mostly tests your measurements: judge from the client, run the validity gate, prove the target got traffic, and treat a pass that hides a large jump as a finding.
• Faults fall into six classes. Gray failures look healthy to some observers; correlated failures defeat redundancy. Both are where the expensive outages live.
• Failure domains share a ladder, but in synchronous training one GPU’s domain is the whole job, and shared dependencies cross every domain.
• Teams over-test easy, total faults and under-test slow, partial, delayed ones: recovery under load, bad configuration, stragglers, checkpoint restore.
• FMEA ranks failure modes by severity, occurrence, and detectability; every unverified mitigation is a hypothesis, severe rows enter the portfolio regardless of RPN, and you re-rank after every finding.
• A hypothesis has a fault, a load, thresholds, and a reason, plus a detection path with a time for gray faults.
• Recent provider outages are mostly propagated config or content meeting a latent defect, and recovery under load; test staged rollout, bake time versus time to symptom, and recovery, not another pod kill.

Sections: One pod, under load · A fault taxonomy, including gray and correlated failures · Failure domains for traditional and AI stacks · Where to focus: pitfalls by architecture · FMEA and the risk-ranked portfolio · Writing a hypothesis · Case studies from recent outages

Chapter 7 ran Kestrel’s first real experiment by hand (Lab 7.1, the INC-2041 cold start) and built a risk-ranked backlog with an AI-assisted FMEA (Lab 7.2). Chapter 6 built the safety layers around any experiment: default deny, the chaos-enablement flag, run permissions, and the kill switch. This chapter turns the hand-run process into an engineered one: prompts that leave no gaps, context the model can trust, a…

• A chaos prompt is a specification. C.H.A.O.S. makes sure it has context, a hypothesis, a precise action, observability, and safety; every fault needs magnitude, scope (including which traffic), duration, and a stop rule.
• The system prompt holds the rules, but it isn’t a security boundary. Every rule that matters is enforced again outside the model.
• The Context Pack is the model’s whole world. Generate it from live sources with read-only rights, never read secrets, record provenance and disagreements, fail on missing data, and include behavior-relevant config such as the simulated warm-up penalty.
• Validate everything deterministically: schema, policy, live checks, then human approval.
• Each layer has blind spots that the next one covers, and every rule needs a negative test and a known-good case that still passes.
• The pipeline gives each step one job, passes state as files, keeps AI away from execution, records which model really answered, never retries a policy denial, and always cleans up.
• Prompts are code. Evaluate with repeated runs and deterministic safety scorers, treat each stack profile as its own model, and remember how little a small sample proves.
• Assume the model will be fooled. Layers that don’t depend on it (validator, approval, account controls, a kill switch that survives a failed stop) make being fooled harmless.

Sections: Anatomy of a chaos prompt (C.H.A.O.S.) · Context engineering · Output contracts, schemas, and policy validation · Prompt chains and pipelines (run controls: Chapter 6) · Testing, evaluating, and versioning prompts; managing model non-determinism · Adversarial safety: prompt injection and poisoned context

Chapter 8 used AI to write automation: a prompt chain, a validator that fails closed on unsafe artifacts (Lab 8.3), an evaluation harness, and red-team cases from Kestrel’s poisoned postmortem. This chapter uses it to make decisions in the chaos lifecycle: what to test, when to stop, where the breaking point is, and what a run means. Every technique reuses Chapter 8′s contracts and validator, and Chapter 6′s rule…

• Match AI autonomy to the cost of a wrong decision. Each level needs its own authorization boundary; move up one level at a time, on evidence.
• Every claimed fix in a postmortem is a hypothesis; Done (unverified) fixes go first, and configuration supplies the rest. Hypotheses cite their source, treat it as untrusted, and name the mechanism.
• Combine abort signals with OR. Missing data aborts for signals that should always publish; sparse counts are zero-filled. Learned guards fail closed, train without chaos windows, are backtested, and never replace static and burn-rate signals.
• Bisection for one parameter, Bayesian optimization for several, with an objective that measures the whole fault. Every trial is a guarded experiment: sequential, bounded, budgeted; the result is a range.
• Agents decide; tools enforce. Start checks approval against a content hash; stop is always allowed and escalates to a kill switch that doesn’t depend on the agent; there’s no generic shell tool.
• AI-written analysis is a draft. Separate facts, mechanisms, and counterfactuals; check numbers in code; don’t rule out per-pod causes with aggregates; close findings with experiments, not tickets.
• Measure whether the AI adds value: verified findings and review time, not the fluency of its plans.

Sections: Where AI fits, and autonomy levels · Hypotheses from incidents and architecture · AI-based steady state and auto-abort · Intelligent fault-space exploration · Agentic chaos operations · AI-assisted analysis and learning loops · Is the AI adding value?

Chapter 9 gave you the AI techniques for chaos: hypotheses from incidents, a composite autoabort, a search for the smallest fault that breaks an SLO, and fact-checked analysis. This chapter opens Part III by pointing them at the lowest layer a service depends on: the CPU time, memory, and storage the kernel and kubelet hand out, planned with Chapter 8′s prompt chain, stopped by Chapter 4′s stop paths, and kept…

• “High CPU” is four failures. Saturation slows every pod on a node; throttling stalls one container while its average looks fine; noisy neighbors and steal take CPU from outside your pod or your VM.
• Requests become weights that share a busy node; limits become quotas that apply even on an idle one. With n runnable threads and a quota of Q ms per 100 ms period, the throttled gap is 100 − Q / n ms.
• Read throttling from cAdvisor’s throttled-period ratio, confirm it in cpu.stat , and check the CPU count runtimes see ( GOMAXPROCS before Go 1.25).
• An OOM kill is the kernel enforcing a container’s limit; the container restarts in place, cold.
• A node-pressure eviction is the kubelet protecting a node; it ranks pods by usage above requests, then priority, and it ignores PodDisruptionBudgets.
• Honest requests are the defense against eviction cascades. Size memory and disk faults from allocatable capacity and the kubelet’s live thresholds, never at 100%.
• Slow storage becomes slow requests through synchronous commits and pools: 10 connections holding a commit for t seconds cap throughput at 10 / t (Little’s law). An injected I/O delay isn’t t; measure it.
• The blast radius of a disk fault is set by its layer: node volume, cloud volume, pod path, or pod storage. On a laptop, never fill a node disk; use a pod’s ephemeral-storage limit.

Sections: CPU: saturation, throttling, noisy neighbors, steal · Memory: evictions and OOM kills · Disk, storage, and I/O

Chapter 10 taught how a single node’s compute, memory, and storage fail, and its Kestrel log found CPU-limit throttling on checkout. This chapter moves outward from the node to everything between services: packets, the connection tables that track them, the names that find them, and the cluster add-ons (DNS, metrics, webhooks, CNI, CSI) that every workload leans on without noticing. Two of Kestrel’s incidents live…

• Small network faults cause large tail effects because TCP waits before it retransmits: at least 200 ms for unrecovered data loss, about 1 s for a lost SYN. Estimate exposure with 1 − (1 − p)ⁿ.
• Conntrack and NAT limits are rate problems: rate × hold time gives occupancy, connection reuse is the fix, and preprod at a tenth of the rate won’t show the wall. ndots:5 makes one external lookup up to eight queries. It doesn’t affect short in-cluster names, so measure before you claim an ndots change helped.
• Pools and caches hide DNS outages from running services and expose them to anything new. A DNS fault with no effect is a valid result only when you’ve proved it was active.
• Choose fault tools by the layer they impair; the mechanism, not the selector, sets the blast radius. A rule that matches pod IPs misses traffic sent to a ClusterIP.
• Asymmetric partitions keep health checks green while requests fail slowly. HTTP/2 and gRPC concentrate fate on long-lived connections; GOAWAY handling and deadlines decide what users see during a failover.
• Running pods survive most add-on failures; deploys, scaling, new connections, and new volumes don’t. Test add-ons with sustained faults, not restarts.
• A webhook’s failurePolicy defaults to Fail . Decide it per rule, and rehearse the breakglass step.
• Check that your stop path doesn’t depend on what you’re breaking, and keep a second stop that does not.
• A scope exception goes in at every layer that enforces scope (validator, admission, fault tool), recorded with an approver, limited to the lab’s label, and reverted by the script that made it.

Sections: Network mechanisms: TCP timers, conntrack and NAT, DNS · Fault tools and the layer each impairs · Asymmetric partitions, gRPC stream resets, and HTTP/2 GOAWAY; long-lived connections during failover · The control plane and cluster add-ons

Chapter 11 tested the network and the cluster services every pod depends on: DNS capacity (INC-2156), a webhook that failed closed (INC-2188), and a metrics-server outage that froze the HPA. This chapter moves down to the nodes, then up to the chain of controllers that adds nodes and pods when load rises, and breaks Kestrel’s slowest incident, INC-2077, into that chain’s segments. Chapter 13 picks up where the chain…

• Node loss is five different events. A drain respects PDBs and grace periods; a graceful OS shutdown respects grace periods only if the kubelet is configured for it; the other three respect neither.
• After abrupt loss, three clocks run: the node-monitor grace period (50 s since Kubernetes 1.32), the 300-second default toleration, and the replacement’s warm-up. Traffic reaches dead endpoints during the first, and those failures happen in the client, where server-side abort alarms can’t see them.
• StatefulSets wait on purpose. The out-of-service taint speeds recovery for a node you know is dead, and causes split-brain on an isolated one.
• A node experiment’s blast radius comes from live placement, includes every pod on the node, and can’t be limited to synthetic traffic.
• Spot notice is 2 minutes on AWS and up to 30 seconds on Azure; Google Cloud gives no dedicated notice by default, only up to 30 seconds of best-effort shutdown. Budget the drain against the shortest window, with no notice as the worst case.
• Test consolidation, drift, upgrades, and architecture moves under load, with a budget that sets the pace and a pre-check that fails closed.
• The scale-up chain is a sum of independent delays. In INC-2077, about 72% of the six minutes was waiting for capacity; even without it, signal lag plus warm-up (about 140 s) exceeded the 90-second rise.
• Negative-priority placeholders take node launch off the critical path for a spike’s first slice, at a constant cost. Pre-scaling and Pending alerts cover the rest.
• Scaling up runs into limits that aren’t CPU. Do the connection math with maxReplicas .

Sections: Five kinds of node loss · Node switches: spot, consolidation, upgrades, architecture changes · Autoscaling: the scale-up chain

Chapter 12 ended with the scale-up chain: a spike arrives, pods go Pending, nodes launch, and new pods start cold (INC-2077). That chain is one way a system gets overloaded. This chapter asks what happens next: does the overload end when the spike or the node loss ends, or does the system keep itself overloaded? You’ll learn to find the line between those two outcomes and to move it. Chapter 14 then applies the idea…

• Big outages are usually one failure plus a feedback loop, and each common loop has a metric signature. In every loop, work that produces no value consumes the capacity needed to produce value. Measure goodput at the client, not throughput at the server, and track amplification per tier against its own baseline.
• Estimate before you inject: Little’s law on checkout’s thread pool shows where its queue forms and why a CPU-based autoscaler may not see it.
• A metastable failure persists after its trigger is removed. Vulnerable systems look healthy, triggers can be small, and recovery needs a stronger intervention than the trigger. INC-2103 and the 2025 AWS and Google Cloud incidents all fit.
• A crude simulation shows the shape: recovery depends on trigger and load, the boundary is sharp, and one retry per request can sustain the loop.
• Map the boundary in preprod only, with response variables fixed in advance, a tested reset between cells, the four look-alike checks, and a total time cap. Report boundary cells as fractions with intervals.
• Apply defenses one at a time. A retry budget moved the boundary; dropping expired work removed it on the model. Adaptive concurrency, shedding with a queue bound below the deadline, and backoff sized to the timeout each break specific parts of the loop.

Sections: Feedback loops and goodput · Metastable failures; a simulation you can run · Finding the tipping point safely · Defenses, one at a time (retry budgets, dropping expired work, adaptive concurrency, load shedding, backoff)

Chapter 13 gave you the metastability map: a transient trigger, load held steady, and a verdict on whether goodput comes back. It also showed that a retry budget moves the boundary and dropping expired work clears it. This chapter applies that method to the largest triggers a cloud system faces: a managed database failing over, a zone disappearing, and a region failing over. These experiments can’t be confined to…

• These are the book’s largest-blast-radius labs: each has a “do not run unless” gate and a reviewed allow-list change, and a writer failover needs a 100% blast-radius exception. Local routes are client-side only.
• Client-observed database failover is the sum of four clocks: service failover, DNS TTL, client DNS cache, and pool eviction. The last can turn a 30-second event into a 30-minute partial outage, and validation on borrow doesn’t stop it.
• Read the client before predicting a cache fault: Kestrel’s fallback costs latency, not database load.
• Check what’s deployed before planning broker faults; FIS can throttle only ec2 and kinesis APIs.
• A zone evacuation needs a live pre-check for placement (requests × 1.2) and throughput (1.5 times normal load per pod). Zonal shift moves traffic, not pods or capacity.
• In the Chapter 13 simulation, losing a zone at 70% load leaves the system stuck after the zone returns; a retry budget helps only for short losses, and dropping expired work recovers every cell.
• Serverless experiments test idempotency, redelivery, and DLQ routing, and must prove the fault applied.
• Measure RPO from acknowledged writes, test switchover and failover separately, and find hidden cross-region dependencies by blocking the path back to the primary.

Sections: Do not run unless · Database failover from the client’s side · Caches, brokers, and API throttling · Zone evacuation · Serverless and event-driven systems · Multi-region disaster recovery

Chapter 14 tested dependencies you don’t run from the client’s side, and Kestrel found that database client DNS caching stretched a failover well past the provider’s own. This chapter turns the same method on security controls, which are dependencies too: each claims that something will be blocked, detected, or revoked within some time. You reuse Chapter 6′s opt-in flags, freeze flag, and kill switch, Section 4.4′s…

• A security control is a hypothesis with a bound; until an experiment exercises it, nobody knows. Its failures are silent: drift, pipeline changes, routing, tuning, and inherited defaults (INC-2290).
• Security chaos measures known controls against benign conditions, repeatedly; it complements penetration tests and red teams, and supports DORA’s TLPT without replacing it.
• Detection is a chain: predict each hop from configuration, add them, then measure. A cloud provider’s log delivery is a hop too (about 5 minutes on average for CloudTrail, per AWS), so each route gets its own bound.
• Simulate the condition, never the attack, on lab-only, flagged resources; revert independently and fail closed; deconflict with the SOC without suppressing alerts.
• Single-event security alerts don’t use for: and must see a counter’s first event. Silence alerts count arrivals. A detection counts only when a receiver gets it.
• Route a security alert on what it is, never on a drill label: a real attacker’s use carries none.
• Chaos tooling is attack surface: prove each “can’t”, pin actions, models, and charts, and keep running the validator’s negative suite.
• Report TTD, TTC, and false negatives against the bound with sample-size caveats, and ATT&CK coverage in three states over an explicit scope.

Sections: Security controls are hypotheses too · What to test · Running security experiments safely · Experiments by layer · Securing the chaos tooling itself · Measuring security resilience

Chapter 15 asked what each security control does when its own dependency fails. This chapter asks it of every dependency: timeouts, retries, breakers, bulkheads, and fallbacks decide how a caller behaves when a callee is down, slow, flaky, or wrong, and each is a claim you can test. Chapter 13 modeled the retry loop that keeps an overload going; this chapter tests it on real clients, then moves “the system” outward…

• Slow is the most dangerous dependency failure; Little’s law says how many threads it holds.
• A fault reaches users through six links. Measure each, and link 1 at the caller.
• Each layer’s worst case (attempts, timeouts, backoff) must be shorter than its caller’s timeout; Kestrel’s is 3,150 ms against 800 ms and 2,000 ms. Deadlines stop abandoned work; a callee’s 499 isn’t proof of one.
• Retries multiply across layers and add most load when failures are correlated. A budget caps amplification at about 1 + budget, at the cost of more failures under independent errors.
• Breakers make fallbacks fast, bulkheads contain one dependency’s slowness, and fallbacks must be correct; choose fail-open or fail-closed per dependency.
• Header-matched faults are confined only if every hop propagates the header and a negative test proves it.
• Clients are part of the system: measure per-action outcomes and client amplification, especially during recovery. Idempotency keys prevent duplicates only when the server stores them.

Sections: The fault is not the symptom: the causal chain · Timeouts, deadlines, retries, and amplification · Circuit breakers, bulkheads, fallbacks · Which experiment proves which control · Header-scoped faults and header propagation · Client-side, mobile, and edge chaos

Chapter 16 followed a fault from a dependency to the user and proved the controls on the call path: timeouts, deadlines, retry budgets, circuit breakers, bulkheads, and header-scoped faults. This chapter goes inside the service and below it: the runtime that executes the code, the pools that ration its connections and threads, the data stores, caches, and queues it relies on, the pipelines that move its data, and…

• Container limits cover the whole process. Budget the heap plus everything else, from committed native-memory figures; checkout’s 60% heap leaves about 205 MiB of its 512 MiB for the rest.
• GC death spirals feed themselves; a cap on work in flight breaks them, a bigger heap only delays them.
• Lab 7.1′s cold start was simulated by WARMUP_PENALTY_MS ; Lab 17.1 measures the real JIT warm-up.
• Where you inject a runtime fault decides which control you test; name the method and the reason.
• Pools fail by Little’s law. Fail fast with short pool timeouts, isolate with bulkheads, bound every queue, and do the fleet-wide connection math (12 × 10 = 120 against a limit of 100, locally).
• Alarms built on completed requests are blind to a hang for the length of the pool timeout, then late, and never see a lock shorter than it; measure at the caller and on in-flight signals.
• Caches protect databases from load they can’t handle; stampedes need coalescing, jittered TTLs, and a database-side limit.
• Messaging needs acks=all with min.insync.replicas , idempotent consumers, a dead-letter path, and duplicates counted by business key.
• Pipelines fail quietly; reconcile per window with counts and checksums. Exactly-once is only as good as the sink.
• Configuration, secrets, certificates, clocks, and backups fail rarely and badly: reject blank config, rotate past a connection lifetime, test renewal and expiry, skew at the leeway boundaries, and time real restores.

Sections: Runtime, heap, and garbage collection · Concurrency and resource pools · Data stores, caches, and messaging (vector databases: Chapter 20) · Batch and streaming pipelines · Configuration, secrets rotation, certificates, time, and backups

Chapter 17 finished the application layer with backups that count only once a restore is proven and a connection pool that fails fast instead of waiting 30 seconds. Both ideas return here in a harsher form: a checkpoint is a backup that must restore under pressure, and a collective is a pool of GPUs in which one slow member sets everyone’s pace. You’ll leave with a measured baseline, a checkpoint interval you can…

• The collective is a barrier, so one GPU’s failure or slowness belongs to the whole job. Measure ETTR.
• Link faults appear first as slow or hung collectives, often without an error.
• One rank’s high compute with everyone else waiting is a straggler; high wait everywhere is the interconnect. GPU utilization alone can’t tell throttled from busy.
• Lost work per failure is about T/2 + R. Young’s T ≈ √(2CM) is throughput-optimal; a deadline, a preemption notice, maintenance, or rollback needs can override it, and at Kestrel’s scale the deadline does.
• Set the collective timeout explicitly and test both sides: too long burns GPUs on hangs, too short kills healthy jobs.
• Checksummed, atomically renamed checkpoints let a job skip damaged ones, but a missing manifest leaves no trace; count orphans, test restore under stress, and keep a rollback point.
• Stragglers don’t fail. Compare ranks, add a job-level step-time band, predict detection time from the window and for: , and measure it.
• Test that GPU health automation takes the catalog’s action per Xid and keeps the node out.
• State each injection method’s limit; catch SDC only with redundancy or replay.

Sections: The training plane: data, GPUs, collectives, checkpoints · GPUs, interconnects (NVLink, InfiniBand, RoCE, EFA), and accelerators (cloud specifics in Appendix B) · What to monitor, and telling bottlenecks apart · Lost work: checkpoint intervals and Young’s approximation · Node loss, NCCL timeouts, and elastic restart · Checkpoint storage and restore · Stragglers and gray failures · GPU errors, node health automation, and silent data corruption

Chapter 18 made Kestrel Assist’s training plane resilient: lost work bounded by the checkpoint interval, restarts proven, and a throttled GPU (INC-2210) caught by step-time outliers. Lab 18.1 also gave you the GPU telemetry this chapter reads. The other half of the GPU fleet serves requests, and it fails differently: a training job loses work, while an inference service makes people wait. This chapter tests that…

• Each layer of the inference plane has its own queue. Before you inject anything, find where requests wait, what limit makes them wait, and what the server does at that limit.
• KV-cache cost per token = 2 × layers × KV heads × head dimension × bytes: 28 KiB for the local 1.5B model, which explains its 41,200-token cache. Long prompts fill the cache before the sequence slots run out.
• Model servers overload quietly: vLLM queues and preempts with recompute, llama.cpp defers behind busy slots, and errors stay low while TTFT rises. Watch waiting, KV use, preemptions, and TTFT; judge by goodput.
• Shed before requests miss their deadline: bound ≈ completion rate × (TTFT budget − prefill), with 429 and a capped Retry-After . Over-length requests get a non-retryable error. Scale on queue size, below the bound.
• Cold start is node, image, weights, engine, and readiness; time each link, and make readiness a real token.
• The gateway owns timeouts, retries, fallbacks, request and token limits, input caps, and hooks. Retry at one layer, label fallbacks, and check streams by content: a 200 can carry a cut stream or a broken body.
• Gang scheduling prevents partial placement; waitForPodsReady bounds how long a stuck job holds quota. Preemption, quotas, capacity, and reservations decide who keeps GPUs; test them through the scheduler’s view of capacity.

Sections: The inference plane: gateway, router, model servers (vLLM, Triton, Ray Serve), KV cache · Overload: KV cache, queueing, preemption, shedding · Scaling inference: signals, cold starts, and model-weight load time · Model gateways, rate limits, and fallbacks · Schedulers, quotas, and GPU capacity

Chapter 19 made Kestrel’s inference plane shed load honestly: under INC-2234′s long-context spike, the server returns 429 with Retry-After instead of queueing until time to first token hits 40 seconds. This chapter moves up one layer, to the applications built on that plane. A RAG service, an agent, and a guardrail all call models, and all of them can turn a slow or failed dependency into a fluent, successful, wrong…

• RAG fails mostly by succeeding: a slow index, an empty or stale result, or a slow embedder still produces a fluent answer. Decide the fallback, label every response, and count fallbacks.
• Compute the time budget from the timeouts first: Kestrel’s nominal worst case before the model is called is 5.5 s; only a request deadline makes it a hard bound.
• Calibrate the empty-result threshold (Kestrel’s 0.3 let an off-topic question through), do the index memory arithmetic (3,072 bytes per 768-dimension float32 vector), and test rebuilds, compaction, hydration, and failover under query load.
• Rebuild beside the serving collection, check recall, then switch an alias. Treat any embedding model change, even one with the same dimension, as a rebuild gated by the retrieval evaluation.
• Bound agent loops in code with iteration, token, tool-call, and deadline budgets; compute worst-case time and tokens; and assert on actions taken, with idempotency keys set by code, not the model.
• Schema drift becomes wrong answers under lax parsing and hand-offs under strict parsing; keep safety rules in code that checks fields.
• A guardrail’s failure mode must be decided in advance, per direction, and proven by timing out the classifier. Count fail-open and fail-closed decisions separately, alert on any fail-open, and alert when blocks fall to zero. Keep the guard’s timeout below its caller’s, or the caller silently decides.
• Measure quality under fault with an evaluation set and a grader that bypasses the fault; report confidently wrong answers and cost per request beside availability and latency.

Sections: RAG under fault: retrieval timeouts, empty or stale results, embedding service degradation · Vector databases under failure · Agents and multi-agent systems · Safety guardrails under stress · Quality under fault: evaluation sets, judge models that bypass the fault, cost per request

Chapters 7 to 20 ran every experiment in preprod or on the local profile, ending with Chapter 20′s RAG, agent, and guardrail faults, where Kestrel Assist’s empty-retrieval answers (INC-2267) and fail-open guardrail (INC-2290) were found and fixed. Every one of those results assumes that preprod behaves like production. This chapter tests that assumption, decides which results transfer, and sets the bar for running…

• Preprod finds breaking points; production confirms defenses at safe magnitude under real conditions. Tipping-point searches, quota exhaustion, and destructive faults never belong in production.
• A header scopes only Layer 7 faults; a pod, node, network, or stress fault’s blast radius is real traffic.
• Diff the Context Packs: compare ratios as well as sizes, resolve endpoints to accounts, treat missing fields as unknown, and block everything on an isolation risk until a re-run diff proves it fixed.
• Local results transfer as mechanisms, not magnitudes: the fix for Lab 7.1′s cold start transfers; its 567 ms doesn’t.
• In preprod you create the load; in production you measure it and add only capped, tagged synthetic traffic.
• Live headroom is u × (R + S) ÷ R × N ÷ (N − k), per resource, from live data, with warm-up counted as removed capacity.
• An off-peak pass is evidence about off-peak: shared limits, in-flight work, recovery, and mix change at peak.
• Production needs a promotion record with residual risk and previous-stage evidence, stage limits, a concurrency cap (Kestrel’s is 10% of real traffic), fixed roles, measured abort latency, and a preflight that fails closed on stale, future-dated, or missing checks. The AI is never an approver.
• When it goes wrong: stop, stop the load, verify the effect, treat it as an incident, restore, review, and restart from preprod.

Sections: Two environments, two jobs · Making preprod results trustworthy · Load in preprod vs production · Earning the right to run in production · When a production experiment goes wrong

Chapter 21 decided where each experiment belongs and on what evidence (Section 21.4′s promotion record). This chapter answers what: the experiments most worth running are the ones your own outages, and other people’s, have already written down. You’ll turn Kestrel’s postmortems into hypotheses and replays, take the INC-2103 story that Chapters 13 and 16 began one step further (a replay of its mechanism on the local…

• A postmortem is a list of claims; give each a status backed by evidence. Only 3 of Kestrel’s 16 Done action items have effectiveness evidence.
• Write a failure contract before a plan. Its amplifiers row is the one most often lost.
• Each contributing factor gets its own test: an experiment, a static check, a simulation, or “fix first”.
• A replay must reproduce the incident without the fix, or a pass with the fix proves nothing; its fidelity map states what’s exact, scaled, substituted, and omitted. Verify the claimed control, not a neighbor: Lab 22.1′s fix run verifies Chapter 16′s timeout fix, not INC-2103′s retry budget.
• External incidents become hypotheses, or written “not applicable” records; near misses show defenses that worked by luck.
• Only verified catalog entries block, with thresholds from verified runs, not the SLO. A gate runs cheapest first, reopens findings as recurrences, gives invalid runs no verdict, and allows a narrow, recorded bypass.
• Track verification coverage, time to verified fix, and recurrence rate, with strict counting rules.

Sections: Every postmortem is a set of hypotheses · Incident → experiment mapping · Contributing factors, not a single root cause · Incident replay · External incidents as free hypotheses · Near misses · Continuous verification: the regression catalog and resilience gates · Program metrics for incident-driven chaos

Chapter 22 turned Kestrel’s postmortems into experiments: it split INC-2301 into contributing factors (Section 22.3), replayed incidents by mechanism (Section 22.4), and made passing experiments into a regression catalog and resilience gates in the pipeline (Section 22.7, the one home for gates). This chapter takes the delivery path itself as the system under test. Progressive delivery is chaos engineering’s closest…

• Canary, A/B, blue/green, shadow, flags, and cells each give a different mix of control, exposure bound, and way back. A control on the same traffic at the same time turns a fault into an experiment.
• Routing isolation confines any fault on the routed pods, including faults a header can’t scope. A fault on a shared dependency is confined only by the header, and needs a negative test to prove it.
• Chaos canaries need fresh, equal-size, simultaneous groups, sticky random assignment, and an A/A check first.
• Judge distributions with a rank test (Kayenta: Mann-Whitney U at 98%), put a tolerance in the hypothesis, keep the judged metrics few, and treat no data as a failure.
• Sample size sets bake time: 0.1% → 0.5% errors needs about 2,900 requests per group; a 3.0% → 2.85% conversion drop needs about 198,000 sessions per cohort, which only production has. A p99 rests on n/100 requests.
• Each stage’s bake must cover time to symptom plus detection plus margin, promotion must wait for a passing verdict, and readiness counts: a pause starts only when its pods are ready.
• A judge must pass good releases too. A healthy canary has no error series, so default the numerator where traffic exists, never the whole ratio.
• Config, content, and data take the same staged path as code; content-hashed config names make each change a canary and keep the rollback target intact.
• Every pipeline dependency gets a recorded fail-open or fail-closed decision, a test, and a measured time to ship a fix while it’s down. Break-glass never depends on what’s being tested.

Sections: Canary, A/B, blue/green, shadow traffic, feature flags, cells · Chaos as a canary: control vs experiment · Chaos on the delivery system · Pipeline dependencies: registry, secrets, flags; fail open or fail closed · A/B cohorts for the cost of degraded modes · Progressive delivery tools

Chapter 23 finished the production part of the book: chaos as a control-vs-experiment canary, and the delivery path as a target. Every experiment so far has tested machines and their automation. This chapter tests the people who answer when those aren’t enough: whether the alert reaches someone, whether the runbook works as written, and whether the on-call role may do what it says. It ends with the game day, where…

• Recovery depends on people as much as systems: test all six links, and time people separately. Routing tests use drill labels, never abort labels.
• Runbooks are programs that run on people: execute them literally, and test every command as the on-call identity.
• Start with tabletops and announced drills; a facilitator can stop any drill. Reviews are blameless or useless, and a runbook fix is verified by a drill.
• Game days need a safety officer with final, unconditional authority to abort (anyone may call a stop; nobody can overrule one), a stop procedure that survives one failed stop, and injects simulated by instruction.
• A second cloud changes the stop paths and the kill switch; a GPU game day earns its cost on decision, gray, and capacity paths.

Sections: Culture and people: enablement, alerts, routing, runbooks, drills, blameless reviews . . 331 24.2 Game days, including multi-cloud and GPU scenarios

Chapter 24 worked on the people: drills, blameless reviews, and a game day that found a runbook step on-call couldn’t run. This chapter turns individual findings into a program that keeps finding weaknesses, turns them into verified fixes, and can prove both to leadership and to an auditor. It collects the evidence of Chapters 2–24 into one quarterly report (Lab 25.1) and writes the business case and 90-day plan…

• Approve by tier in preprod and by stage in production, in the pipeline; tier 4 is game day only.
• Keep a short governance set, and decide every finding’s state from its evidence.
• Measure weighted coverage of recorded runs, time to verified fix with its n, recurrence, invalid run rate, and cost per verified fix; never experiments run.
• Build the business case from your own incidents in their own units, and say so when lost revenue alone doesn’t cover the cost.
• Start at S1 × A1 and move one maturity axis at a time on evidence.
• Chaos supplies compliance evidence, not compliance.
• A leadership report fits on one page and traces every number to an export.

Sections: Governance: tiers, artifacts, the life of a finding, metrics · How organizations adopted chaos engineering: case studies · Making the case: ROI, sponsorship, and the first 90 days · A chaos maturity model and adoption roadmap · The cost of chaos, including GPU economics · Compliance evidence: DORA, UK operational resilience, and for AI systems the EU AI Act and NIST AI RMF · Reporting to leadership

Chapter 25 turned single experiments into a program whose 90-day plan ends with “report and expand”. This chapter is that expansion, run by you and graded. Each project combines labs you’ve done under one set of controls, and you defend the result. It hands off to Appendices A (contracts), B (cloud mappings), and G (lab status).

• A capstone is a small chaos program graded on what you can prove, under six common requirements.
• Each run.sh proves the controls, runs the experiments, and re-runs the fix with --fixed ; you apply the fix, and --change records it.
• Correct local plans may fail validate.sh ; record the violation, never rename the action.
• T_opt = sqrt(2 × C × M), with M for the whole job (about m/N); a deadline can set a shorter interval.
• Bake ≥ time to symptom + detection time + margin (Section 23.3).
• A safety failure caps the total at 50%; an unverified fix caps experiment design at Good.

Sections: How the capstones work · Project 1: Kestrel Commerce under load (Core) · Project 2: Kestrel Assist under failure (AI infrastructure) · Project 3: From outage to production canary (Production & multi-cloud) · Grading, with bands, and how safety failures cap the score · A sample submission, in outline · Capstone defense: presenting results and answering review-board questions

The contracts and templates every lab in this book uses, in one place. This appendix is the canonical source. Where a chapter shows a contract or prompt, it shows an excerpt of the version here, and if they ever differ, this appendix wins.

• The contracts and templates every lab in this book uses, in one place.
• This appendix is the canonical source.
• Where a chapter shows a contract or prompt, it shows an excerpt of the version here, and if they ever differ, this appendix wins.

Sections: A.1 The C.H.A.O.S. prompt anatomy · A.2 SC-1: the Safety Contract · A.3 CP-1: the Context Pack · A.4 OC-1: the Output Contract · A.5 G-1, V-1, and A-1: the standard prompts · A.6 Experiment card template · A.7 RR-1: the run record · A.8 The chaos-enablement flag schema

Dated. Every fact in this appendix was checked against provider documentation or the sources named in “Sources, known gaps, and revision schedule” at the end of this appendix in October 2026 (research date 22 October 2026); rows that rest on secondary sources say so. Cloud services change monthly; Azure Chaos Studio in particular is mid-way through a re-platforming. Before you rely on a limit, a notice time, or a…

• Every fact in this appendix was checked against provider documentation or the sources named in “Sources, known gaps, and revision schedule” at the end of this appendix in October 2026 (research date 22 October 2026); rows that rest on secondary sources say so.
• Cloud services change monthly; Azure Chaos Studio in particular is mid-way through a re-platforming.
• Before you rely on a limit, a notice time, or a preview status, open the named source and check it for your Region. [verify] marks a statement this edition’s sources didn’t confirm; treat it as a lead, not a fact.
• How to use this appendix The chapters teach each fault on the Local and AWS routes in full, and give Azure and Google Cloud as binding notes: what changes, what has no equivalent, and the workaround.

Sections: B.2 Kubernetes and node provisioning; GPUs, interconnects, capacity; parallel file systems · B.3 Load-testing services; monitoring and GPU telemetry; pipeline identity · B.4 Fault-action equivalence table by fault class · B.5 Index of Azure and Google Cloud lab files in the repository · B.6 Chaos tool reference: what each tool injects, how it stops, and its equivalent fault actions

This appendix, the prompt pilot report and the system prompt library, is in the full edition. Its system prompts are in the companion repository, in toolkit/contracts/prompts/ and each lab’s prompts/ folder.

• This appendix, the prompt pilot report and the system prompt library, is in the full edition.
• Its system prompts are in the companion repository, in toolkit/contracts/prompts/ and each lab’s prompts/ folder.

Sections: Prompt Pilot Report and System Prompt Library

This appendix, additional exercises with solutions, is in the full edition.

• This appendix, additional exercises with solutions, is in the full edition.

Sections: Additional Exercises, with Solutions

This appendix, careers and the job market: what postings ask for, roles and levels, a skills matrix, and chapters mapped to roles, is in the full edition.

• This appendix, careers and the job market: what postings ask for, roles and levels, a skills matrix, and chapters mapped to roles, is in the full edition.

Sections: Careers and the Job Market

Terms are defined as this book uses them. The reference in parentheses is where the term is explained first or most fully. Acronyms are listed under the acronym, with the expansion in parentheses. Where two terms are easily confused, a “Compare” note points to the other. A-1 The standard Analyze prompt, run after every experiment. It checks run validity first, builds a timeline, and separates observed facts from…

• Terms are defined as this book uses them.
• The reference in parentheses is where the term is explained first or most fully.
• Acronyms are listed under the acronym, with the expansion in parentheses.
• Where two terms are easily confused, a “Compare” note points to the other.

Sections: Glossary

This appendix, lab status per route, recorded runs, fault-action permissions, and troubleshooting by symptom, is in the full edition. The data behind it is in the companion repository: each lab’s status is in its lab.yaml and in catalog/labs.yaml , and its recorded runs are under outputs/ .

• This appendix, lab status per route, recorded runs, fault-action permissions, and troubleshooting by symptom, is in the full edition.
• The data behind it is in the companion repository: each lab’s status is in its lab.yaml and in catalog/labs.yaml , and its recorded runs are under outputs/ .

Sections: Lab Verification and Traceability

• The interview question bank, by topic and level, is in the full edition.

Sections: Interview Question Bank

This appendix, the local AI lab stack reference: services, models, the gateway, the fault proxy, and troubleshooting, is in the full edition. The stack’s own guide is in the companion repository, in platform/ai-stack/README.md .

• This appendix, the local AI lab stack reference: services, models, the gateway, the fault proxy, and troubleshooting, is in the full edition.
• The stack’s own guide is in the companion repository, in platform/ai-stack/README.md .

Sections: The Local AI Lab Stack Reference