ITIL Monitoring and Event Management: Events, Alerts, Thresholds and Observability
The best incident is the one users never notice. Monitoring and event management systematically observes services and service components and records and reports selected changes of state identified as events. It provides the early warning that lets teams act before an outage, and the evidence they need to diagnose one quickly.
Key Terms
| Term | Meaning |
|---|---|
| Event | Any change of state that has significance for the management of a service or CI |
| Monitoring | Repeated observation of a service or component to detect events |
| Alert | A notification that an event needs attention |
| Threshold | A value that, when crossed, triggers an event or an alert |
Types of Events
- Informational: normal and useful for records, such as a backup completed or a user logged in. Usually no action.
- Warning: something is approaching a limit, such as disk usage at 80%. Action may be needed soon.
- Exception: something has gone wrong or a threshold has been breached, such as a service stopped. Action needed now.
What to Monitor
| Layer | Examples of measures |
|---|---|
| Infrastructure | CPU, memory, disk, network traffic, hardware health |
| Applications | Response times, error rates, queue length, job failures |
| Services | Availability as users see it, such as a synthetic login test |
| Business | Orders per minute, payments completed, tickets created |
| Security | Failed logins, unusual access, malware alerts |
Monitoring from the user’s point of view is the most valuable. A server may be running while the login page returns an error; a synthetic test that logs in every minute catches that.
From Event to Action
- Detection: a tool notices a change of state.
- Filtering: the system ignores noise and duplicates.
- Correlation: related events are grouped (fifty alerts from one failed switch become one).
- Classification: the event is judged informational, warning or exception.
- Response: automatic action (restart a service), an incident, a change or a notification to an on-call engineer.
- Review: check that the rules worked and adjust thresholds.
Alert Fatigue
If a team receives hundreds of alerts a day, they start ignoring them, and the one that matters is missed. Good practice includes:
- Alert only on conditions that need a human response.
- Tune thresholds based on real behaviour, not default values.
- Group related alerts and suppress alerts during planned maintenance.
- Route alerts to the team that can act, with a clear runbook link.
- Review noisy alerts monthly and delete or fix them.
From Monitoring to Observability
Monitoring tells you when something you predicted goes wrong. Observability is the ability to understand what is happening inside a system from the data it produces: logs, metrics and traces. In complex, distributed systems, you cannot predict every failure, so teams collect rich data to ask new questions when something unexpected happens. The two are complementary: monitoring for known conditions, observability for exploration.
Worked Example
A retailer’s monitoring shows disk usage on a log server at 82%, a warning event that creates a low-priority ticket. The team clears old logs and sets up automatic rotation. A week later, a synthetic checkout test fails twice in a row, an exception event that pages the on-call engineer and opens a P2 incident. Correlation shows that the payment gateway is slow rather than the retailer’s servers, so the engineer contacts the supplier. Users see a short delay but no failure, and the team has the timeline to discuss in the supplier review.
Common Mistakes
- Monitoring everything but acting on nothing.
- Using default thresholds that fit no real system.
- Only monitoring servers, not the user experience.
- Failing to link alerts to runbooks and owners.
- Never reviewing alert rules after the system changes.
- An event is a significant change of state; monitoring detects events and alerts people or systems.
- Classify events as informational, warning or exception and act only where needed.
- Monitor from the user’s point of view, and correlate events to reduce noise.
- Observability complements monitoring by helping teams explore unexpected failures.