Designing Game Days & Chaos Experiments

Recap: From Experiment Templates to Exercises

Foundations You'll Need Today

1. Why Game Days Are on the Exam

The SAP-C02 exam tests design judgment under constraint, and one of the most reliable constraints it uses is the gap between what an architecture claims and what it has actually been proven to do. A team can draw a three-AZ, multi-region diagram with automatic failover and still have a recovery time objective that exists only in a slide deck. The exam's resilience domain is built around that gap, and Game Days are the mechanism AWS documents for closing it. When a question describes a workload with a stated RTO, a documented runbook, and no evidence the runbook has ever been executed, the correct answer is almost always to test the recovery path rather than to add more redundancy.

This maps most directly to the Resilient Architectures domain, but it also touches Operational Excellence, because a Game Day is fundamentally an operations-as-code practice: the exercise is scripted, the success criteria are quantified in advance, and the output is a set of corrective actions rather than a vague sense that things went fine. The Well-Architected Reliability pillar names failure management as one of its four areas, and within failure management the guidance is explicit that recovery procedures must be tested, not merely written. A question that offers "increase backup frequency" or "add a second standby region" as an answer to an untested-runbook scenario is testing whether you understand that more infrastructure does not substitute for validated procedure.

There is a second, subtler exam angle. Game Days are the point where several earlier topics converge: the composite alarms from Day 29 determine whether the exercise produces signal or noise, the synthetic canaries from Day 30 determine whether you detect the failure from the customer's perspective, and the FIS stop conditions from Day 33 determine whether the exercise stays safe. A scenario that asks how to validate an SLO end to end is often really asking whether you would run a controlled failure and measure the detection and recovery path, rather than whether you would add another dashboard.

2. How a Game Day Actually Runs

A Game Day is a scheduled, time-boxed exercise in which a team deliberately introduces a realistic failure into a system and observes whether the documented detection and recovery mechanisms behave as designed. The word "scheduled" carries weight: unlike an ad hoc chaos experiment run by one engineer to see what happens, a Game Day has a defined start and end, named participants, a written hypothesis, and pre-agreed success criteria. The hypothesis is the part teams skip and the part that makes the exercise useful. "We believe that if we lose one Availability Zone, the Application Load Balancer will drain the affected targets within 60 seconds, the Auto Scaling Group will replace capacity in the surviving zones, and customer-facing error rate will stay below 0.5% for the duration" is a hypothesis. "Let's kill an AZ and see what happens" is not.

Mechanically, the exercise runs in four phases. In the planning phase the team selects a failure mode, writes the hypothesis, defines the success criteria as measurable thresholds, identifies the stop conditions that will abort the run, and notifies anyone whose systems might be affected. In the execution phase the failure is injected — typically through an FIS experiment template, though a Game Day can also be run by manually failing over a database or revoking a credential — while participants watch the same dashboards an on-call engineer would watch. In the observation phase the team records what actually happened against what was predicted: time to detection, time to first alarm, time to automated recovery, time to manual intervention if any, and the peak customer-visible impact. In the retrospective phase the deltas between prediction and reality become backlog items.

The critical structural property is that the failure injection is bounded by the same stop conditions that govern any FIS experiment. If the error-rate alarm breaches its threshold, the experiment halts automatically rather than waiting for a human to notice. This is what makes it defensible to run a Game Day against production, which is where the exercise has value. A Game Day run only in a staging environment validates the runbook against staging's topology, staging's data volume, and staging's traffic patterns — none of which match production. The exam consistently favors testing in production with bounded blast radius over testing in an environment that does not reproduce the failure mode.

3. The Core Decision Boundary: What Are You Trying to Prove?

Every Game Day design question reduces to a single fork: is the exercise validating a detection path, a recovery path, or a decision path? The three are different exercises with different failure modes, and conflating them produces an exercise that proves nothing. A detection exercise asks whether the monitoring stack notices the failure and pages the right person within the target window. A recovery exercise asks whether the automated or documented remediation actually restores service within the RTO. A decision exercise asks whether the humans in the loop make the right call — fail over or wait, escalate or absorb — under time pressure and incomplete information.

The fork matters because the success criteria are completely different. A detection exercise can succeed even if recovery is slow, as long as the alarm fired and the page reached the on-call engineer. A recovery exercise can succeed even if detection was late, as long as the system returned to health within the RTO. A decision exercise is the hardest to score because the "correct" call depends on information that may not have been available at the time, and the retrospective has to evaluate the decision against the information the responder actually had rather than against the outcome.

In practice most Game Days target recovery, because that is where the RTO claim lives and where the gap between documentation and reality is widest. But the exam will present scenarios where the real weakness is detection — a workload with excellent automated failover and no alarm that fires when it happens — and the correct answer is to exercise the detection path first. The table below maps the fork to its observable signals.

4. Scoping the Exercise: Blast Radius, Environment, and Timing

The knobs that determine whether a Game Day is safe and useful are blast radius, environment, and timing, and each trades safety against fidelity. Blast radius is the fraction of the fleet or the number of targets the injection touches. A single-task termination proves that the scheduler replaces a task; it does not prove that the service survives losing a third of its capacity. A full AZ failure proves the latter but consumes real capacity and real customer impact. The right setting is the smallest blast radius that still exercises the mechanism you are testing — if the hypothesis is about AZ-level failover, you need AZ-level scope, and no smaller injection will validate it.

Environment is the second knob, and the tradeoff is stark. Staging is safe and reproducible but rarely reproduces production's data volume, traffic distribution, dependency graph, or configuration drift. Production is faithful but carries real risk, which is precisely why the stop conditions matter. The defensible position, and the one the exam rewards, is to run in production with a bounded blast radius and automated abort conditions, and to reserve staging for rehearsing the mechanics of the injection itself before the real exercise. Teams that only ever test in staging tend to discover during a real incident that the runbook's assumptions about production topology were never true.

Timing is the third knob and the one most often mishandled. Running a Game Day during peak traffic maximizes fidelity but also maximizes the cost of a mistake. Running it at 3 a.m. minimizes customer impact but also means the on-call engineer who would respond to a real incident is asleep, so the exercise tests the automated path only and says nothing about the human path. The common compromise is to run during business hours with the full response team present and available, accepting a small amount of customer-visible risk in exchange for testing the complete detection-to-recovery chain including the humans. The table below summarizes the tradeoffs.

5. Sizing the Exercise and the Numbers That Bound It

Game Day sizing is really RTO and RPO arithmetic made concrete. If a workload's stated RTO is fifteen minutes, the exercise must be long enough to observe whether recovery completes inside that window, which means the observation period has to extend past the RTO rather than stopping the moment the injection ends. A common mistake is to declare success when the alarm clears, without measuring the full path from injection to steady-state health. The measurement that matters is wall-clock time from the moment the failure is introduced to the moment the service is serving normal traffic at normal error rates, and that number is what gets compared against the RTO.

The FIS experiment itself is bounded by its stop conditions, and those thresholds should be set tighter than the SLO breach point, not equal to it. If the service's error-rate SLO allows 1% errors over a five-minute window, the stop condition should trip well before that — the exercise is meant to reveal weakness, not to consume the error budget. Similarly, the blast radius percentage in the target selection should be the minimum that exercises the mechanism. FIS supports targeting by tag, by resource ID, and by percentage of a matched set, and the percentage mode is the one that makes an exercise reproducible across a fleet whose size changes between runs.

Duration is bounded by the failure mode. Fast failures — instance termination, task kill, AZ failover — complete their recovery within minutes and the exercise can be short. Slow failures — memory leaks, connection pool exhaustion, disk fill, certificate expiry — need the exercise to run long enough for the symptom to develop, which can mean hours. The table below lists the failure modes most commonly exercised and the observation window each requires.

6. Failure Modes of the Exercise Itself

The most common way a Game Day fails is that it proves nothing because the injection never actually reached the system under test. A target selection that matches zero resources, an IAM role that lacks permission to perform the action, or an experiment scoped to a tag that no longer exists all produce a run that reports success while nothing happened. The symptom is an exercise where every metric stayed flat and the retrospective concludes the system is resilient, when in fact the failure was never introduced. The first diagnostic move is always to confirm the injection landed: check the FIS experiment's action log and verify the target count is what you expected before interpreting any other signal.

The second failure mode is the opposite — the injection lands but the stop conditions do not fire when they should, because the alarm they reference is misconfigured, evaluating the wrong metric, or in INSUFFICIENT_DATA state. An alarm that has never had enough datapoints to leave INSUFFICIENT_DATA cannot trigger a stop condition, which means the safety net is absent precisely during the exercise that needs it. Before any Game Day, the stop-condition alarms should be verified as being in OK state with a healthy evaluation history, not merely present in the template.

The third failure mode is a recovery that works but for the wrong reason. If the service recovered because a human manually scaled the fleet rather than because the Auto Scaling Group did it, the exercise has validated the human, not the automation, and the runbook's claim about automated recovery is still unproven. Distinguishing these requires the observation phase to record who or what performed each recovery action, which is why the exercise should be run with the response team watching rather than silently. The table below maps symptoms to first diagnostic moves.

7. The SRE Angle: Game Days as Error-Budget Spend

From an SRE perspective a Game Day is a deliberate expenditure of error budget in exchange for information, and it should be budgeted and scheduled like any other planned risk. The exercise consumes some fraction of the service's allowed unreliability, and the return is a measured recovery time, a validated runbook, and a list of corrective actions. Framing it this way makes the go/no-go decision tractable: if the service has ample error budget remaining, the exercise is cheap; if the budget is nearly exhausted, the exercise should be deferred or run at a smaller blast radius, because the team cannot afford the additional unreliability on top of whatever is already consuming the budget.

The observability requirements for a Game Day are the same as for an incident, which is a useful forcing function. If the team cannot tell during the exercise whether the service is healthy, they will not be able to tell during a real incident either, and that discovery is itself a valuable output. The minimum instrumentation is a customer-facing availability signal (synthetic canary or load balancer error rate), a latency signal at the percentile that matters, a saturation signal for the constrained resource, and an alarm on each that is known to be in OK state before the exercise begins. Composite alarms are particularly useful here because they let the exercise's stop condition require correlated signals rather than tripping on a single noisy metric.

The runbook shape that emerges from a well-run Game Day is a decision tree rather than a checklist. The exercise reveals the branch points — at what error rate do we fail over, at what latency do we shed load, who has authority to declare the incident — and those branches get written down with the thresholds that were validated. The retrospective output is not "the system worked" but a set of specific deltas: the alarm fired four minutes later than predicted, the runbook's failover step referenced a console page that has moved, the connection pool needed a manual restart that no step mentioned. Each delta becomes a backlog item with an owner.

8. Edge Cases and Exam Gotchas

The single most-tested gotcha is the substitution trap: a scenario describes an untested recovery procedure and offers "increase backup frequency," "add a second standby region," or "enable Multi-AZ" as answers. None of these validate the procedure. More redundancy changes the recovery characteristics but does not prove the runbook works, and the exam expects you to select the answer that tests rather than the answer that adds. A related trap offers "run the exercise in a staging environment" as the safe choice; it is safer but it does not reproduce production's failure mode, and the exam generally prefers bounded production testing.

The second gotcha concerns stop conditions. A question may describe an FIS experiment with no stop condition, or with a stop condition referencing an alarm that has never fired, and ask what the risk is. The answer is that the experiment has no automated abort and will continue injecting failure regardless of impact, which is exactly the scenario Day 33's stop-condition discipline exists to prevent. A third gotcha is the difference between a Game Day and a chaos experiment run ad hoc: the Game Day's defining features are the pre-written hypothesis, the quantified success criteria, and the scheduled participation of the response team. An experiment without those is a test, not a Game Day, and the exam distinguishes them.

Finally, watch for scenarios that conflate the exercise with the remediation. A Game Day that reveals a gap does not fix the gap; the corrective actions do. An answer choice that says "run a Game Day to resolve the issue" is wrong if the issue is a known defect — the Game Day would only confirm what is already known. The exercise is for discovering unknown gaps and validating claimed capabilities, not for closing defects you have already identified.

9. Game Days vs. the Practices They Get Confused With

Game Days sit in a family of resilience practices that are easy to conflate: chaos experiments, load tests, disaster recovery drills, and penetration tests. The distinctions matter on the exam because each answers a different question and produces a different artifact. A chaos experiment is the injection mechanism — it answers "what happens if I break this." A Game Day wraps a chaos experiment in a hypothesis, a schedule, and a response team, and answers "does our documented recovery actually work." A load test answers "does the system hold up under expected and peak traffic," which is a capacity question rather than a failure question, though the two are often run together.

A DR drill is the closest relative and the most commonly confused. A DR drill exercises the failover to a standby region or environment and is typically a larger, less frequent, more disruptive event. A Game Day is usually scoped to a single failure mode within a single region and can be run far more often. The relationship is that Game Days build the muscle memory and validate the components that a DR drill then assembles into a full failover. A penetration test is unrelated in purpose — it probes for exploitable vulnerabilities rather than for recovery capability — and should not be substituted for resilience testing.

The practical rule is to pick the practice that matches the claim you need to validate. If the claim is "we recover from an AZ loss in under fifteen minutes," run a Game Day with an AZ-scoped injection. If the claim is "we can operate from our secondary region," run a DR drill. If the claim is "we handle Black Friday traffic," run a load test. The table below makes the selection explicit.

Hands-On Lab: A Full AZ-Failure Game Day (60 min)

Scenario Question Drills (20 min)

Peek into Tomorrow

Sources