Runbook — Communications Outage (Template)

Trigger

• Alarm fires for a critical component (example: “Communications Loss”).
• Or: multiple related alarms within 5 minutes.

Safety checks (do first)

1. Confirm you are working in the correct environment (site/system).
2. Check for any “Do Not Operate” status or active maintenance window.
3. If there is any safety risk, stop and escalate immediately.

Scope triage (avoid chasing ghosts)

1. Check whether multiple sites are affected (widespread vs single-site).
2. Confirm monitoring platform is healthy (status page / internal alerts).
3. Record last known good telemetry time.

Triage steps (low-risk first)

1. Record timestamp + alarm text + any related alerts.
2. Re-poll / refresh telemetry once.
3. Check dependencies (power state, upstream connectivity indicator).
4. If permitted: perform a single low-risk recovery action and monitor.

Decision points

• If multiple sites are affected → escalate as platform/network issue.
• If single site remains offline > 10 minutes → escalate per matrix.
• If any sign of equipment damage/unsafe state → stop actions and escalate.

Escalation

• Escalate to: on-call engineer / field tech / manager (based on severity).
• Provide: scope, last known good time, steps attempted, current status.

Summary:
- Trigger/alarm:
- Scope (single site vs widespread):
- Last known good telemetry:
- Checks completed:
- Actions attempted (low-risk first):
- Current status:
- Safety concerns:
- Requested next steps / owner:

Verification

1. Confirm telemetry stable for 10 minutes.
2. Confirm alarms cleared and no repeated comms alarms.
3. Update log/ticket with actions and outcome.

Closeout notes (recordkeeping)

• What happened (1–2 sentences)
• What you did (bullets)
• Who you notified
• What still needs follow-up

Changes

• v0.1 — Initial template published.