CloudWatch Synthetics & Canaries for SLO Monitoring

Recap: From Alarm Logic to User-Visible Truth

Foundations You'll Need Today

1. Why This Is on the Exam

SAP-C02 scenario questions are written around a specific kind of gap: the architecture looks healthy on every dashboard the team has built, yet the business is losing money. The exam's Domain 2 (Design for New Solutions) and Domain 4 (Continuous Improvement for Existing Solutions) both probe whether you can identify what a monitoring strategy is blind to, and Synthetics is the canonical answer whenever the blind spot is the user's actual experience rather than the server's internal state. A question that describes a multi-region active-active application with comprehensive CloudWatch dashboards, X-Ray tracing, and composite alarms, and then asks what additional monitoring would detect a regional edge failure before customers call the support line, is pointing at canaries.

The reason this is a durable exam topic rather than a trivia item is that it sits at the intersection of two things the exam cares about deeply: SLO definition and blast-radius reasoning. An availability SLO expressed as "99.9% of checkout attempts succeed" cannot be measured by any metric your application emits, because your application only knows about the requests that reached it. Requests that died at the DNS resolver, at the CDN, or in the client's browser never appear in your logs at all. Synthetics closes that measurement gap by generating known-good traffic on a schedule and asserting on the result, which turns an unmeasurable SLO into an observable one. Expect questions that give you an SLO target and ask which monitoring approach can actually attest to it.

There is also a cost-and-complexity dimension the exam likes to test. Canaries are not free, they are not instantaneous, and they cannot cover every path. A scenario that asks you to monitor a 40-step internal admin workflow with a canary is testing whether you understand that canaries are best reserved for high-value, stable, externally reachable user journeys. The exam will offer you a canary as a distractor for problems that are better solved with distributed tracing, log-based metrics, or synthetic load testing, and it will offer you those alternatives as distractors for problems that only a canary can solve. Knowing the boundary is the point.

2. How a Canary Actually Runs

A canary is not a ping. It is a small Node.js or Python script that CloudWatch Synthetics executes on a schedule inside an AWS-managed, serverless execution environment. The script receives a Synthetics runtime library that wraps Puppeteer (for browser-based canaries) or raw HTTP clients (for API canaries), and it is expected to do three things: drive the target, assert on what it observes, and emit structured results. The runtime handles the rest — launching the headless browser, capturing screenshots and HAR files on failure, uploading artifacts to S3, and publishing metrics into the CloudWatch namespace CloudWatchSynthetics.

The execution model matters for reasoning about cost and reliability. Each canary run is an isolated invocation with its own ephemeral environment; there is no persistent browser session between runs, which means every run pays the cold-start cost of launching Chromium. That cost is why the minimum useful interval is generally measured in minutes rather than seconds, and why a canary that takes 45 seconds to execute at a 1-minute frequency is effectively running continuously. The runtime also enforces a timeout, and a canary that exceeds it is recorded as a failure with a distinct status code rather than a generic error — a distinction that matters when you are writing alarms, because a timeout usually means the target is slow rather than down.

Results flow out on two channels. The first is metrics: every run publishes SuccessPercent, Duration, Failed, and 2xx/4xx/5xx counts, plus any custom metrics your script emits via the Synthetics library. The second is artifacts: screenshots, logs, and HAR files written to the S3 bucket you nominate when you create the canary. The metrics are what your alarms consume; the artifacts are what a human consumes at 3 a.m. when the alarm fires and they need to know whether the page rendered a login form or a CloudFront error page. Designing a canary without thinking about the artifact side is a common mistake — the metric tells you something broke, the screenshot tells you what.

Finally, canaries can run either on a schedule or on demand, and they can be triggered by other systems. The scheduled mode is the default and the one exam scenarios describe. On-demand invocation is useful for validating a deployment immediately after it completes, and for the "smoke test" step in a pipeline. Both modes publish to the same metrics, which means an alarm cannot distinguish a scheduled failure from a pipeline-triggered one unless you tag the runs — worth knowing when you are designing a deployment gate that should not page the on-call engineer.

3. The Core Decision Boundary: What Deserves a Canary

The single fork every Synthetics scenario question hinges on is whether the thing you need to monitor is observable from inside your infrastructure. If it is, a canary is redundant and expensive. If it is not, a canary is the only tool that can see it. The practical test is to ask where the failure would occur: if the failure would occur in code you deploy and instrument, internal metrics and traces will catch it; if the failure would occur in the path between the user and your code — DNS, CDN, load balancer configuration, TLS certificates, third-party dependencies, client-side rendering — only an outside-in probe will catch it.

This boundary has a second dimension that is easy to miss: stability. A canary is a scripted assertion, which means it encodes an expectation about what the target should look like. If the target's UI changes weekly, the canary becomes a maintenance burden that fails for reasons unrelated to availability, and the team learns to ignore its alarms. The exam will sometimes present a fast-moving internal tool as a canary candidate and expect you to recognize that the maintenance cost outweighs the benefit. Canaries earn their keep on stable, high-value, externally reachable journeys: login, search, checkout, the primary API endpoint a partner integrates against.

The table below is the decision boundary in its most compressed form. Read it as a set of rules for eliminating distractors rather than as a taxonomy to memorize.

4. Configuration Modes and Their Tradeoffs

Once you have decided a canary is warranted, the configuration knobs determine what it costs and how quickly it detects a problem. The first and most consequential is the blueprint. Synthetics ships pre-built blueprints for the common cases — Heartbeat Monitoring (a simple URL fetch and status assertion), API Canary (a scripted HTTP request sequence with response validation), Broken Link Checker (crawls a page and verifies every link), Visual Monitoring (compares a screenshot against a baseline and flags pixel differences), and GUI Workflow Builder (records a browser session and replays it). Choosing a blueprint is choosing how much of the assertion logic you write yourself. Heartbeat is nearly free to configure and catches availability failures; GUI Workflow catches functional regressions but requires you to maintain a recorded script that breaks whenever the UI changes.

The second knob is frequency, and it is where the cost conversation actually happens. A canary running every minute costs roughly sixty times what the same canary running every hour costs, and the detection latency for a failure is bounded below by the frequency. The right frequency is a function of your SLO: if your availability SLO is 99.9% measured monthly, a failure that lasts five minutes consumes about 0.7% of your monthly error budget, so a one-minute canary that detects it in under two minutes is proportionate. A five-minute canary on the same SLO would let a single incident consume several percent of the budget before anyone noticed. The exam rarely asks you to compute this, but it does ask you to reason about detection latency versus cost.

The third knob is the runtime version and the script itself. Synthetics runtimes are versioned (for example syn-nodejs-puppeteer-7.0 and later), and AWS deprecates older runtimes on a published schedule. A canary pinned to a deprecated runtime will eventually stop running, which is a silent monitoring failure — the worst kind, because the absence of alarms looks like health. The fourth knob is the S3 artifact location and retention, which determines how much forensic evidence you have after an incident and how much you pay for storing screenshots. The fifth is the VPC configuration: a canary can run inside a VPC to reach private endpoints, but doing so requires the canary's execution role to have ENI permissions and the subnets to have a route to the target, and it removes the canary's ability to test the public path. Most user-facing canaries should run outside the VPC precisely because the public path is what you are trying to verify.

5. Sizing, Limits and Quotas

The numbers that matter for Synthetics fall into three groups: execution limits, cost drivers, and integration limits. On execution, each canary run has a configurable timeout with a default of 30 seconds and a maximum of 15 minutes, and the runtime itself imposes a hard ceiling on how long a single run can occupy its environment. A canary whose script routinely approaches the timeout is a canary that will produce false failures under load, so the practical guidance is to keep the scripted path short and assert early rather than driving a long workflow.

On cost, the two drivers are the number of runs and the duration of each run. Synthetics pricing is per canary run, with a separate charge for the browser-based runs that use more memory. This is why the frequency decision from the previous section is the dominant cost lever: doubling the frequency doubles the bill, and a canary that takes 90 seconds to execute costs more per run than one that takes 15 seconds. The S3 artifacts add a smaller, usually negligible storage cost, but a canary that captures a full HAR file on every run rather than only on failure can accumulate meaningful storage over months.

On integration, the limits that bite in practice are the CloudWatch metrics retention (standard resolution metrics are retained for 15 months, but the canary's own success metrics are published at one-minute resolution and are best alarmed on with a short evaluation period) and the alarm evaluation window. A canary running every five minutes publishes a data point every five minutes; an alarm with a one-minute period and a single evaluation will see missing data for four out of five minutes and may enter INSUFFICIENT_DATA. The correct pattern is to set the alarm period to match the canary frequency and to treat missing data as breaching, because a canary that stops running is itself an incident. The table below collects the numbers worth remembering.

6. Failure Modes and What They Look Like in Production

The most dangerous canary failure is not a red alarm; it is a canary that has quietly stopped running. This happens when the runtime version is deprecated, when the execution role's permissions are revoked, when the S3 artifact bucket is deleted, or when the canary's schedule is disabled during a maintenance window and never re-enabled. In every case the symptom is identical: the SuccessPercent metric stops receiving data points, and an alarm configured with the default missing-data treatment may sit in INSUFFICIENT_DATA indefinitely without paging anyone. The first diagnostic move when you suspect this is to check the canary's last run timestamp in the Synthetics console, not the alarm state.

The second failure mode is the false positive caused by target drift. A GUI Workflow canary that asserts on a specific button label will fail the moment a designer renames the button, and the failure will look exactly like an availability incident. The distinguishing signal is that the canary's Duration metric stays normal while SuccessPercent drops — a real outage usually shows both degrading, because a slow or unreachable target takes longer to fail. When you see success drop with duration flat, the first thing to check is the screenshot artifact, which will show the new UI rather than an error page.

The third failure mode is the canary that is too aggressive and becomes a load source. A canary that drives a full checkout flow every minute from three regions is generating real orders and real payment authorizations, and if the target is not idempotent you have created a business problem. The mitigation is to design the canary against a dedicated test account or a sandbox path, and to make the scripted actions idempotent where possible. The fourth failure mode is credential expiry: a canary that logs in with a stored credential will start failing the day that credential rotates, and the failure will be indistinguishable from an authentication outage unless you monitor the canary's own error messages. Storing canary credentials in Secrets Manager and having the script fetch them at runtime is the pattern that avoids this.

7. The SRE Angle: Canaries as SLO Instruments

From an SRE perspective, a canary is not primarily a monitoring tool; it is a measurement instrument for an SLO that would otherwise be unmeasurable. The distinction matters because it changes how you configure it. If you are using a canary to page an on-call engineer, you want it to be sensitive and you accept some false positives. If you are using it to compute an availability SLO, you want it to be representative of real user traffic, which means the canary's success rate should track the real user success rate closely enough that you can defend the number in a post-incident review. Those two goals pull in different directions, and mature teams often run two canaries: a fast, sensitive one for alerting and a slower, more representative one for SLO reporting.

The alarm design follows from the SLO. A canary publishing SuccessPercent every minute should have an alarm on SuccessPercent < 100 with a period of one minute and an evaluation of one or two datapoints, treating missing data as breaching. That alarm should feed into the composite alarm structure from Day 29 rather than paging directly, so that a single canary failure in one region does not wake someone up when the other regions are healthy. The composite pattern — canary failure AND elevated real-user error rate — is the one that distinguishes a genuine outage from a canary-specific problem, and it is the pattern the exam is most likely to describe.

The runbook shape for a canary alarm is short and specific. First, open the screenshot artifact from the failing run; it answers the question "what did the user see" faster than any log query. Second, check whether the failure is isolated to one region or global, which distinguishes a regional edge problem from an application problem. Third, check the canary's own last successful run timestamp to rule out the silent-stop failure mode. Fourth, if the screenshot shows a valid page, check whether the assertion itself is stale. That four-step sequence resolves the large majority of canary alarms, and writing it down before the first incident is what turns a canary from a noise source into a useful signal.

8. Edge Cases and Exam Gotchas

The gotchas cluster around three themes: what a canary cannot see, what it costs to run, and how it interacts with the rest of the observability stack. On the first theme, a canary cannot see the experience of a real user on a slow connection, cannot see a failure that only affects one browser, and cannot see a failure that only affects authenticated users unless you script the authentication. It also cannot see a failure that occurs between the canary's region and the user's region if the canary runs from a single region — which is why multi-region canaries are the pattern for multi-region applications. A single-region canary on a multi-region application will report success while an entire region's users are down.

On the second theme, the cost trap is frequency. A team that sets a canary to run every minute "to be safe" and then adds a second canary for each of ten critical paths has created a monitoring bill that can rival the application's compute bill. The exam will sometimes present this as a scenario where the correct answer is to reduce frequency or consolidate paths rather than to add more canaries. The related trap is artifact retention: capturing a full HAR file on every successful run rather than only on failure is a common misconfiguration that quietly accumulates storage cost.

On the third theme, the interaction gotchas are about alarm wiring. A canary alarm with a one-minute period on a five-minute canary will flap between OK and INSUFFICIENT_DATA, and if missing data is treated as not-breaching the alarm will never fire. A canary alarm that pages directly rather than through a composite will generate noise during regional degradations. A canary that runs inside a VPC cannot test the public path, so a VPC-configured canary on a public-facing application is testing the wrong thing. And a canary whose execution role lacks s3:PutObject on the artifact bucket will fail to publish artifacts, which means the alarm fires but the screenshot is missing — a failure mode that is invisible until you need the evidence.

9. Synthetics vs. the Services It Gets Confused With

Synthetics is most often confused with three things: Route 53 health checks, X-Ray tracing, and load testing. The confusion with health checks is the most common, because both are outside-in probes that run on a schedule. The difference is depth. A health check asks "does this endpoint return a healthy status code" and nothing more; it cannot log in, cannot follow a redirect chain, cannot assert on page content, and cannot measure the time to interactive. A canary can do all of those things. The exam will offer a health check as the answer to a question about detecting a broken login flow, and the correct answer is a canary, because the health check would return 200 from the load balancer while the login page renders an error.

The confusion with X-Ray is about direction. X-Ray traces a request that already reached your application, breaking it into per-hop latency and identifying which internal service is slow. A canary generates a request from outside and measures the total time to a successful assertion. They are complementary: a canary tells you the user-visible path is broken, and X-Ray tells you which internal hop is responsible. The exam will sometimes describe a scenario where the canary is failing and ask what to do next, and the correct answer is to use X-Ray to find the slow hop — not to add more canaries.

The confusion with load testing is about volume. A canary is a single synthetic user; a load test is thousands. A canary detects availability and functional regressions; a load test detects capacity limits. The exam will offer a canary as the answer to a question about validating that a system can handle a traffic spike, and the correct answer is a load testing tool. The table below is the disambiguation in its most compressed form.

Hands-on Lab: A Checkout Canary with an SLO Alarm (45 min)

Scenario Question Drills (20 min)

Peek into Tomorrow

Sources