Server Outage Plan
This runbook defines how the IT Department responds to storage node and region-level outages so customer data stays durable and recoverable at every severity level.
Always confirm the affected node against the item list before taking any action — restarting the wrong instance can trigger unnecessary re-allocation and customer-facing downtime.
Severity Levels
| Severity | Description | Escalation |
|---|---|---|
| SEV-1 | Customer data unavailable or at risk of loss — Immediate response | Page on-call lead + Director of Infra |
| SEV-2 | Single node or region degraded, redundancy intact — < 15 min response | Page on-call engineer |
| SEV-3 | Non-customer-facing service degraded — < 1 hour response | Slack #infra-oncall |
| SEV-4 | Cosmetic or monitoring-only issue — next business day | File a ticket |
Response Steps
- Acknowledge the alert in PagerDuty within the target response time for its severity.
- Identify the affected node in the item list and confirm its region and role.
- Check replication status — for hot-tier nodes, confirm the warm/cold replica is serving reads.
- Open an incident channel and post updates every 15 minutes until resolved.
- Write a postmortem within 48 hours for any SEV-1 or SEV-2 incident.
Relationships
Depends on / points to
No outgoing relationships.
Referenced by
Nothing points to this item.
Credentials
Simulated for demo purposes — secrets are encrypted at rest (AES-256-GCM), masked by default, and every reveal is logged with who and when.
No credentials recorded for this item.