CloudWatch Metrics, Alarms & Composite Alarms

Recap: From Data Selection to Signal Selection

Foundations You'll Need Today

1. Why This Is on the Exam

CloudWatch alarms appear on SAP-C02 in two distinct guises, and confusing them is the most common way candidates lose points on otherwise straightforward questions. The first guise is as a component inside a larger architecture: an alarm that triggers an Auto Scaling policy, an alarm that acts as a stop condition on a Fault Injection Service experiment, an alarm that drives a Route 53 health check or a Lambda-based remediation. In those questions the alarm is not the subject of the question — it is a required piece of plumbing, and the exam is testing whether you know that the alarm must exist and what it must be attached to. The second guise is as the subject itself: a scenario describes alert fatigue, a noisy pager, or a monitoring gap, and the correct answer is a specific alarm configuration rather than a different service.

The reason this topic carries weight is that it sits at the intersection of the Resilient Architectures domain and the Operational Excellence concerns that thread through every scenario. A design that has no alarms is not merely incomplete; it cannot satisfy a stated RTO, because nothing detects the failure that starts the recovery clock. A design that has too many alarms is also wrong, because the exam increasingly tests whether you understand that an alerting system which pages on every transient spike is functionally equivalent to one that pages on nothing — the on-call engineer stops trusting it. Composite alarms and anomaly detection bands exist precisely to answer that second failure mode, and they are the two features most likely to appear as the distinguishing detail between two otherwise plausible answers.

There is also a quieter reason. CloudWatch is the default integration point for nearly every other AWS service's health signal, which means a question about ECS task health, RDS storage, Lambda error rates, or NAT Gateway port exhaustion is often really a question about whether you know which metric to alarm on and at what granularity. The exam does not expect you to memorize every metric name, but it does expect you to know the shape of the answer: which namespace, which statistic, which period, and which of the alarm states you would act on.

2. How Metrics and Alarms Actually Work

A CloudWatch metric is a time-ordered series of data points identified by a namespace, a metric name, and a set of dimensions. Dimensions are the part that trips people up, because they are not labels in the Kubernetes sense — they are part of the metric's identity. A metric published as AWS/EC2 with dimension InstanceId=i-abc123 is a different metric from the same name with dimension InstanceId=i-def456, and there is no query that aggregates across them unless you explicitly ask for it. This matters for alarms because an alarm is bound to exactly one metric (or one metric math expression), and the dimensions you choose determine whether that alarm watches a single instance, a single target group, or an aggregate.

Data points arrive at a resolution determined by the publisher. Most AWS service metrics are standard resolution, published at one-minute granularity, and retained for fifteen months with progressively coarser aggregation as they age. Custom metrics can be published at high resolution, down to one-second granularity, and high-resolution alarms can evaluate on those shorter periods. The retention and resolution rules are worth internalizing because they bound what an alarm can possibly detect: an alarm whose period is five minutes cannot react to a thirty-second spike, no matter how the threshold is set, because the spike is averaged into the five-minute datapoint before the alarm ever sees it.

An alarm evaluates a statistic over a period against a threshold, and it does so repeatedly over a number of evaluation periods. The DatapointsToAlarm and EvaluationPeriods pair is the mechanism that controls sensitivity: with three evaluation periods and a datapoints-to-alarm of three, the metric must breach on three consecutive periods before the alarm transitions to ALARM. With three evaluation periods and a datapoints-to-alarm of one, a single breaching period is enough. This is the knob that most candidates overlook, and it is frequently the difference between a design that flaps and one that does not.

Alarms have three states, not two. OK and ALARM are the obvious pair, but INSUFFICIENT_DATA is a real state that occurs when the metric has no datapoints for the evaluation window — a newly created alarm, a metric that stopped publishing because the resource was terminated, or a sparse custom metric. Treating INSUFFICIENT_DATA as equivalent to OK is a genuine production hazard, because a resource that has stopped reporting entirely will sit in INSUFFICIENT_DATA and never page anyone. The TreatMissingData setting exists to let you choose whether missing data is treated as breaching, not-breaching, or ignored, and the correct choice depends on whether absence of data means "healthy and idle" or "dead."

3. The Core Decision Boundary: Single-Metric vs Composite

The fork that most scenario questions hinge on is whether a single metric threshold is sufficient to describe the condition you actually care about, or whether the condition is inherently a conjunction of signals. A single-metric alarm answers a narrow question: is this one number outside its expected range? That is the right tool when the metric is a direct proxy for the thing you care about — disk space consumed, queue depth, replication lag. It is the wrong tool when the thing you care about is a user-visible symptom that can be caused by several different underlying conditions, because then any single metric will either fire on conditions that are not actually user-impacting, or fail to fire on conditions that are.

Composite alarms exist to express that conjunction. A composite alarm takes a rule expression over the states of other alarms — ALARM(latencyAlarm) AND ALARM(errorRateAlarm) — and enters ALARM only when the expression evaluates true. The important architectural property is that composite alarms do not evaluate metrics at all; they evaluate alarm states. That means they compose recursively, and it also means they inherit the evaluation semantics of their children, including the children's own periods and datapoints-to-alarm settings. A composite alarm cannot be more sensitive than the alarms feeding it.

The practical consequence is that composite alarms are the correct answer whenever a scenario describes alert fatigue, a noisy pager, or a requirement to page only on genuine incidents. They are also the correct answer when a scenario describes a condition that is only meaningful in combination — high latency alone might be a slow dependency, but high latency together with elevated error rate is a service degradation. The table below lays out the boundary.

4. Configuration Modes and Their Tradeoffs

Once you have decided what condition you are describing, the configuration knobs determine how faithfully the alarm tracks it. The first and most consequential choice is the statistic. Average is the default and the most misleading for latency, because a service where ninety-nine percent of requests complete in fifty milliseconds and one percent take ten seconds has an average that looks healthy. Maximum catches the tail but is hypersensitive to a single outlier. Percentile statistics — p90, p99, p99.9 — are the correct choice for latency SLOs, and the exam expects you to reach for them when a scenario mentions tail latency or a small fraction of slow requests. The tradeoff is that percentiles require enough datapoints to be statistically meaningful, so they are a poor fit for low-traffic metrics.

The second choice is the period, which sets the resolution of the evaluation. A shorter period detects faster but is noisier, because a single bad minute can breach a one-minute alarm. A longer period is smoother but introduces detection latency proportional to its length. The interaction with the statistic matters: a one-minute maximum is far more volatile than a five-minute average, and choosing both a short period and a maximum statistic produces an alarm that will flap on any transient blip. The usual production pattern is to pair a percentile statistic with a period long enough to be stable, then use evaluation periods to require persistence.

The third choice is the threshold type. Static thresholds are simple, auditable, and correct when the metric has a known acceptable range. Anomaly detection bands replace the static number with a model that learns the metric's normal pattern — including daily and weekly seasonality — and alarms when the metric falls outside a configurable number of standard deviations from the predicted band. The tradeoff is legibility: a static threshold can be explained in a runbook in one sentence, while an anomaly band's behavior depends on training data and can be difficult to reason about during an incident. Anomaly detection is the right answer when the metric's baseline genuinely moves with traffic, and the wrong answer when the metric has a hard physical limit that a static threshold expresses more clearly.

The fourth choice is the action. An alarm can notify an SNS topic, trigger an Auto Scaling policy, stop or start an instance, or invoke an SSM automation. The design question is whether the action is a notification or a remediation. Remediation actions should be paired with a notification, because an automated action that fires silently is indistinguishable from a bug. And any alarm that drives an automated remediation should have a corresponding alarm on the remediation's own success, or you will eventually discover that your self-healing system has been failing to heal for weeks.

5. Sizing, Limits and Quotas

The numbers that matter for this topic are mostly about resolution and retention, because those bound what an alarm can detect. Standard-resolution metrics are published at one-minute granularity. High-resolution custom metrics can be published at one-second granularity, and alarms on those metrics can evaluate with periods as short as ten seconds or thirty seconds. The practical implication is that sub-minute detection requires a custom metric — you cannot get a ten-second alarm on a standard EC2 metric no matter how you configure it.

Retention follows a tiered schedule. Datapoints at one-minute resolution are retained for fifteen days, five-minute datapoints for sixty-three days, one-hour datapoints for four hundred and fifty-five days, and beyond that the data is aggregated further. This matters for capacity planning and for post-incident review: if you want to compare this month's latency against the same month last year, you are looking at hourly aggregates, not the original minute-level data.

Alarm evaluation has its own limits. Each alarm evaluates one metric or one metric math expression, and metric math lets you combine up to a bounded number of metrics into a derived series — useful for ratios like error rate as a percentage of total requests, which is a far better alarm target than raw error count. Composite alarms can reference other composite alarms, but the rule expression has a bounded size, so deeply nested hierarchies eventually hit a ceiling. In practice the useful depth is two or three levels: leaf alarms on individual metrics, a middle layer expressing service-level conditions, and a top layer expressing customer-visible impact.

6. Failure Modes and What They Look Like in Production

The first failure mode is the alarm that never fires because the metric stopped publishing. This is the INSUFFICIENT_DATA trap, and it is the most dangerous because it is silent. A Lambda function that has been deleted, an EC2 instance that has been terminated, a custom metric publisher that has crashed — all of these produce a metric that simply stops, and an alarm with the default missing-data treatment will sit in INSUFFICIENT_DATA indefinitely without notifying anyone. The symptom is a monitoring gap discovered during an incident, and the first diagnostic move is to check the alarm's state history rather than its current state. The fix is to set TreatMissingData to breaching for metrics where absence means failure, and to add a separate alarm on the publisher's own health.

The second failure mode is the flapping alarm. This presents as a pager that fires and clears repeatedly over a short window, and it is almost always a configuration problem rather than a real instability. The usual causes are a period too short for the metric's natural variance, a maximum statistic on a noisy metric, or a threshold set at the edge of the normal range. The diagnostic move is to pull the metric's history over the last week and look at where the threshold sits relative to the distribution — if the threshold is inside the normal band, the alarm is measuring noise. The fix is to widen the period, switch to a percentile statistic, or require multiple evaluation periods.

The third failure mode is the alarm that fires correctly but on the wrong thing. This happens when the alarm watches an infrastructure metric that is a poor proxy for the user-visible symptom. CPU utilization on a fleet behind a load balancer is the classic example: the fleet can be at thirty percent CPU while every request is failing, because the failure is in a downstream dependency. The symptom is an incident where the alarms that fired were not the alarms that mattered, and the diagnostic move is to trace backward from the user-visible symptom to the metric that would have detected it earliest. The fix is usually to add a composite alarm at the service level rather than to remove the infrastructure alarms.

The fourth failure mode is alarm fatigue itself, which is a failure mode of the monitoring system rather than of the application. It presents as engineers acknowledging pages without investigating, or muting alarms during incidents. It is caused by a pager whose signal-to-noise ratio has degraded, and the fix is structural: composite alarms to express conjunctions, anomaly detection for metrics with moving baselines, and a periodic review of which alarms actually resulted in action. An alarm that has never once led to a human doing something is a candidate for deletion or for demotion to a dashboard.

7. The Operational and SRE Angle

From an SRE perspective, alarms are the mechanism that converts an SLO into an actionable signal. An SLO is a target — ninety-nine point nine percent of requests succeed within three hundred milliseconds over a thirty-day window — and the error budget is the amount of failure that target permits. The alarm's job is to fire when the burn rate against that budget is high enough that a human needs to intervene before the budget is exhausted. This framing produces a different alarm design than the infrastructure-centric one: instead of alarming on CPU, you alarm on the ratio of slow requests to total requests, and you alarm on the rate at which the error budget is being consumed rather than on the absolute error count.

Multi-window burn-rate alerting is the pattern that follows from this. A fast-burn alert uses a short window and a high burn rate to catch acute outages — the kind that would exhaust a monthly budget in hours. A slow-burn alert uses a longer window and a lower burn rate to catch chronic degradation that would exhaust the budget over days. Both are needed, because a single window either misses slow burns or pages too often on fast ones. In CloudWatch terms this is implemented with metric math over the request and error counts, producing a burn-rate series that a static threshold alarm can watch.

The runbook shape follows directly. Every alarm that pages a human should have a runbook entry that names the metric, the threshold, the likely causes in order of probability, and the first diagnostic command for each. The runbook should also state what the alarm does not tell you — an alarm on elevated error rate does not distinguish between a code defect and a dependency outage, and the runbook should say so explicitly so the responder does not waste the first five minutes assuming the wrong cause. Alarms whose runbook entry is empty are, in practice, alarms that will be muted during the next incident.

Finally, the alarm configuration itself should be version-controlled. Alarms created by hand in the console drift, and the drift is invisible until an incident. Defining alarms in CloudFormation, Terraform, or CDK means the threshold, the period, and the missing-data treatment are all reviewable in a pull request, and it means a deleted alarm is restored by the next deployment rather than discovered missing during an outage.

8. Edge Cases and Exam Gotchas

The single most-tested gotcha is that composite alarms evaluate alarm states, not metrics. A question that asks you to reduce alert noise by combining two conditions is pointing at a composite alarm; a question that asks you to alarm on a ratio of two metrics is pointing at metric math on a single alarm. These are different features and the exam distinguishes them. If the scenario describes two independent conditions that must both be true, it is composite. If it describes one derived number, it is metric math.

The second gotcha is the INSUFFICIENT_DATA state. Any question that describes a resource being terminated, a publisher failing, or a metric going silent is testing whether you know that the default behavior does not page. The correct answer is almost always to set the missing-data treatment explicitly, and to add an alarm on the publisher itself.

The third gotcha is the statistic. A scenario that mentions a small percentage of users experiencing slow responses is describing a tail-latency problem, and the correct statistic is a percentile, not an average. A scenario that mentions a hard capacity limit is describing a maximum or a sum, not an average. The exam rarely names the statistic in the question; it describes the symptom and expects you to infer it.

The fourth gotcha is the relationship between alarms and automated actions. An alarm that triggers an Auto Scaling policy is not the same as an alarm that notifies a human, and a scenario that requires both will have an answer that configures both. Similarly, an alarm used as a stop condition on a Fault Injection Service experiment must be a real alarm with a real threshold — the exam will sometimes offer a plausible-sounding but non-existent "stop condition" resource as a distractor.

The fifth gotcha is the difference between an alarm and a health check. Route 53 health checks are a separate mechanism with their own evaluation logic, and while an alarm can be used as a health check input, they are not interchangeable. A scenario about DNS failover is asking about health checks; a scenario about paging a human is asking about alarms.

9. This vs. the Services It Gets Confused With

CloudWatch alarms are frequently confused with three other mechanisms, and the exam exploits the confusion. The first is CloudWatch Logs metric filters, which derive a metric from log events rather than from a published metric. A metric filter is the right tool when the signal you care about only exists in log content — a specific error string, a stack trace, a business event — and the resulting metric can then be alarmed on normally. The distinction is that a metric filter is a metric source, not an alarm type.

The second is EventBridge rules, which react to events rather than to metric thresholds. An EventBridge rule is the right tool for reacting to a discrete occurrence — an EC2 instance state change, an API call recorded by CloudTrail, a scheduled tick — while an alarm is the right tool for reacting to a continuous measurement crossing a boundary. A scenario that describes reacting to a state change is asking about EventBridge; a scenario that describes reacting to a threshold breach is asking about alarms.

The third is AWS Health and Trusted Advisor, which surface account-level and best-practice findings rather than metric conditions. These are complementary rather than competing: a Trusted Advisor check might tell you that an alarm is missing, but it does not replace the alarm.

Hands-on Lab: A Composite Alarm That Pages Only on Real Incidents

Scenario Question Drills

Peek into Tomorrow

Sources