App Insights & alerting

Everything Class Twenty-Eight collected watched from outside: the platform's view of your resources — CPU numbers, blob operations, job executions. What it cannot see is the inside of the application: which HTTP request was slow, which database call it was waiting on, which exception fired on which line. Application Insights is Azure Monitor's instrument for that interior view — wire it into the storefront (for many platforms, without touching code; for the rest, a small OpenTelemetry package and a connection string) and the app begins narrating its own execution into the same workspace family you already query.¹

App Insights

Application performance monitoring from inside the code: every request, every outbound dependency call, every exception, correlated into end-to-end traces — so "the checkout is slow" becomes "the checkout spends 1.8 of its 2.1 seconds waiting on this one database query."

Four telemetry types carry most of the value, and they interlock. Requests: every call your app served, with duration and result. Dependencies: every call your app made — SQL, storage, third-party APIs — which is where most slowness actually lives. Exceptions: every error, with stack trace. Traces: the app's own log lines, stitched to the request that produced them. The stitching is the product: one slow checkout request expands into its dependency calls and exceptions as a single correlated story, and the application map draws the whole system — services, dependencies, failure rates on every edge — from that telemetry automatically. Class Twenty-Nine's skills apply unchanged; these are just richer tables.

All telemetry so far shares a blind spot: it reports what happened to traffic that arrived. If DNS breaks, the certificate expires, or the front door falls off the internet, the app records nothing wrong — no requests, no errors, a peaceful dashboard over a dead site. The fix is a customer who never sleeps: an availability test, which requests your URL on a schedule from multiple Azure regions around the world and records success, failure, and response time from each.

The design details are small but load-bearing. A standard test checks more than "did it respond": expected status code, an optional content match string (a page that returns 200 with an error message in it is still down, as far as a customer cares), TLS certificate validity, and — quietly one of the most valuable lines in Azure — a proactive certificate lifetime check that fails the test before the cert expires, converting the classic Saturday-outage into a Tuesday ticket. Run it from at least five locations, and alert when several fail at once — the recommended threshold is locations minus two — so one region's network hiccup does not page anyone, but a real outage, visible from three continents, does.² That multi-location logic is the whole trick: the robot customer distinguishes "the site is down" from "the internet had a moment," which is exactly the judgement a single ping never could.

Detection without delivery is a diary. Azure separates the two cleanly: an alert rule is scope + condition (a metric threshold, a failed availability test, or a Class Twenty-Nine log query on a schedule), and an action group is a named, reusable list of destinations — email, SMS, push, a webhook into Teams or Slack, an ITSM ticket, even a Logic App or Azure Function that does something. Rules reference action groups, many-to-one, so "who gets told" is maintained in one place instead of inside forty rules:

# one action group, reused by every alert that pages on-call
az monitor action-group create \
  --name ag-campux-oncall \
  --resource-group rg-campux-platform \
  --short-name oncall \
  --action email oncall-primary [email protected]

Two disciplines make the machinery trustworthy. First, severity is a promise, not a mood. Azure's levels run Sev0 (critical) to Sev4 (verbose); a team must define what each means in response time — Sev0 wakes a human now, Sev2 is business-hours work, Sev4 notifies nobody — and then honour it, because the day a Sev0 is ignored, the scale means nothing. Second, route by severity, not by alert: the page-someone group, the Teams-channel group, the log-it-only group — three action groups, and every rule picks one by the promise its severity makes. Alerts are also state-based where it matters: an availability alert fires once when the site goes down and once when it recovers, not every fifteen minutes in between — a mercy you will appreciate by §4.

Here is where monitoring programs actually die. Not from missing tools — from a pager that cried wolf. The failure sequence is always the same: enthusiasm creates forty alerts; half fire on conditions that need no action; on-call learns, correctly, that most alerts mean nothing; the learning generalises; and eighteen months later a real Sev0 scrolls past unread in a channel nobody has muted only because muting felt like admitting it. The team then buys a new monitoring tool, and the cycle restarts with better dashboards. Alert fatigue is not a personnel problem. It is a signal-to-noise problem, and it is designed in or designed out.

Every alert is a claim on someone's attention.

Actionable

There is something a human should do upon receiving it — and the alert says what, ideally with a runbook line. "FYI, CPU was briefly high" is not an alert; it is a chart that belongs on a dashboard, per §5.

Owned

A named person or rotation answers it. An alert routed to a twelve-person alias is owned by nobody — everyone assumes someone else is looking, and everyone is right.

Rare

It fires only when the condition is real: thresholds tuned to actual behaviour (not defaults), state-based so one incident is one notification, and reviewed on a cadence where anything that fired without requiring action gets retuned or deleted — the same ratchet as Class Twenty-Nine's query pack, run in reverse.

The test is brutal and worth applying to every rule you ever create: would you wake a colleague at 3am for this, in person, standing at their bed? If yes, it is a Sev0 or Sev1 with a pager. If no — and for most "alerts" the honest answer is no — it is a dashboard panel, a weekly report, or nothing. Teams that keep three meaningful alerts outperform teams with forty ignored ones, in every incident, forever.

What happens to everything that failed the 3am test? It becomes glanceable instead of interruptive — and here too there is a discipline separating useful from decorative. The decorative kind is familiar: a wall of sixteen charts, assembled in week one and understood by nobody by month three, whose actual function is looking impressive behind the team during video calls. The useful kind is built backwards from questions. Before adding any panel, name the question it answers and who asks it: Is checkout healthy right now? (on-call, hourly). Did last night's batch run? (the storefront team, each morning). Is ingestion cost drifting? (the platform lead, monthly). If no one asks the question, the panel does not earn the pixels.

Azure's tool for this is the workbook — an interactive document mixing prose, parameters, and live Class Twenty-Nine queries, closer to a runbook with charts than to a TV wall. The strongest pattern wires the layers of this class together: the alert tells you something is wrong; the workbook it links to shows the affected service's requests, dependencies, and recent deploys side by side — the first three investigation queries pre-run before the engineer has poured coffee. Alerts interrupt; dashboards answer; workbooks investigate. Keep the three jobs separate and each stays good at its own.