Alert Management Policy
1. Objective
This policy defines the requirements for managing the complete lifecycle of alerts - from configuration and classification, through the response SLA and escalation, to the calibration and elimination of alerts that do not fulfil their purpose.
An alert without a response deadline is just noise. A system with hundreds of active alerts that nobody responds to is worse off than a system without alerts - it has created the illusion of monitoring without the substance. Alert fatigue is one of the most documented causes of undetected incidents: when everything alerts, nothing alerts. Effective alert management is a disciplined practice of signal/noise ratio, not of accumulating rules.
The objective of this policy is to ensure that:
- Each alert has a defined and tested response SLA
- Each alert has an associated runbook that guides the response
- Automatic escalation acts when the SLA is exceeded
- Alert quality metrics are tracked and used for calibration
- Alert fatigue is monitored and actively corrected
2. Scope
This policy applies to all security and operational alerts configured in the organisation's monitoring systems, SIEM, APM and on-call platforms.
3. Alert classification
All alerts must be classified by severity, which determines the response SLA and the notification channel:
| Severity | Criterion | First-response SLA | Channel |
|---|---|---|---|
| P1 - Critical | Immediate impact on production or confirmed security compromise | ≤ 5 minutes | PagerDuty/OpsGenie (phone call) + Slack |
| P2 - High | Significant impact on production or high-severity security anomaly | ≤ 15 minutes | PagerDuty/OpsGenie + Slack |
| P3 - Medium | Service degradation or anomaly that requires attention but does not cause immediate impact | ≤ 60 minutes | Slack + ticket |
| P4 - Low | Informational event with potential for degradation if ignored | ≤ 4 hours (working hours) | Ticket |
4. Mandatory runbooks
Each configured alert must have an associated runbook that guides the response:
| Minimum runbook content | Description |
|---|---|
| What this alert detects | Clear description of the event or condition |
| How to verify (diagnosis) | Concrete steps to confirm whether it is a true positive |
| Normal vs anomalous context | What normal behaviour is, to distinguish it from the anomalous |
| Immediate actions | What to do in the first 5 minutes |
| Escalation | To whom to escalate and under what conditions |
| Relevant links | Dashboard, logs, incident playbook if applicable |
Runbooks without the minimum fields or not reviewed for more than 6 months are treated as out of date, and the alert is a candidate for review.
5. Automatic escalation
When the first-response SLA is exceeded without any recorded action, the alert must be escalated automatically:
| Level | L1 | L2 | L3 |
|---|---|---|---|
| Automatic escalation on SLA exceeded | Not applicable | Mandatory (to Tech Lead) | Mandatory (to Tech Lead + AppSec/GRC) |
| Notification to a manager when a P1 persists > 30 minutes | Not applicable | Recommended | Mandatory |
| Escalation to GRC/Compliance and the DPO when there is an indication of a regulatory track (Policy 32 §4.1) | Mandatory (≤ 1 h) | Mandatory (≤ 1 h) | Mandatory (≤ 1 h) |
The escalation chain must be documented and tested periodically (at least once every six months).
6. Prevention of alert fatigue
Alert fatigue is an operational risk that must be monitored and actively managed:
6.1 Alert quality metrics
The following metrics must be tracked monthly:
| Metric | Description | Target |
|---|---|---|
| True positive rate (TPR) | % of P1/P2 alerts that correspond to real incidents | > 70% |
| False positive rate (FPR) | % of P1/P2 alerts that are noise | < 30% |
| Mean response time | Mean time between the alert and the first recorded action | ≤ SLA per severity |
| Alerts without action in 24h | Number of P3/P4 alerts without action in 24h | < 10% |
| Total alert volume per day | Indicator of systemic noise | Downward or stable trend |
6.2 Intervention criteria
If quality metrics worsen systematically:
- Review of the thresholds of the alerts with the highest false positive rate
- Suspension of alerts with TPR < 30% until recalibration
- Aggregation of correlated alerts into a single event
- Removal of alerts that have never given rise to action in 90 days
7. P1 alerts outside working hours (on-call)
For L2/L3 systems, there must be an on-call rotation that guarantees 24/7 response availability for P1 alerts:
- On-call rotation documented and known to the team
- Primary on-call and secondary on-call (backup) defined for each period
- On-call tool configured with the escalation chain
- On-call compensation and limits established in accordance with the HR policy
8. Periodic review of the alert inventory
| Level | Cadence | Scope |
|---|---|---|
| L1 | Annual | Existing alerts; removal of obsolete alerts |
| L2 | Half-yearly | Existing alerts + quality metrics + runbooks |
| L3 | Quarterly | Alerts + metrics + runbooks + escalation simulation |
The review must result in:
- Removal or reworking of alerts with TPR < 30%
- Updating of out-of-date runbooks
- Addition of alerts for newly identified risks
9. Responsibilities
| Role | Responsibility |
|---|---|
| DevOps / SRE | Configure and maintain alerts; manage on-call; track quality metrics |
| AppSec Engineer | Define security alerts; validate security runbooks; calibrate thresholds |
| On-Call | Respond within the SLA; record the action taken; escalate when necessary; provide feedback |
| Tech Lead | Ensure that the team has up-to-date runbooks; manage escalations received |
| GRC / Compliance | Audit SLA compliance; verify the escalation chain; reports |
10. Review and audit of this policy
This policy must be reviewed annually or after any of the following events:
- An incident in which the alert was not responded to within the SLA
- A period of documented alert fatigue (TPR < 50% for > 30 days)
- A change of on-call tool or SIEM
11. Normative and technical references
| Reference | Relevance |
|---|---|
| SbD-ToE Ch. 12 - Monitoring and Operations | US-03, US-10: alerts with SLA, validation and tuning |
Security Monitoring Policy (30_policy-monitorizacao-seguranca.md) | Definition of critical events to alert on |
IRP Integration Policy (32_policy-irp.md) | Integration of alerts with incident response |
| PagerDuty / OpsGenie | Reference on-call and alert management tools |
| Site Reliability Engineering (Google SRE Book) | Alert Philosophy, SLO-based alerting |
| NIST SP 800-61 | Computer Security Incident Handling - detection and analysis |