Purpose
This guide describes the recommended process for identifying, managing, resolving, and reviewing production incidents.
An incident is any unexpected event that affects system availability, performance, security, data integrity, or user experience.
1. Detect and validate the incident
An incident may be reported through monitoring alerts, support requests, customer feedback, or team observation.
Confirm:
- Which system or service is affected
- When the problem started
- Whether the issue is still occurring
- Which users or business functions are impacted
- Whether the problem is isolated or widespread
Record the initial observations and timestamps.
2. Assess severity
Classify the incident based on business and technical impact.
Consider:
- Number of affected users
- Availability of critical functionality
- Data loss or corruption
- Security or privacy implications
- Financial or regulatory impact
- Availability of a workaround
The incident owner should communicate the severity and revise it if the impact changes.
3. Assign responsibilities
For significant incidents, assign the following roles:
- Incident lead: Coordinates the response and makes decisions.
- Technical lead: Directs investigation and remediation.
- Communications owner: Provides updates to stakeholders.
- Recorder: Maintains the timeline, actions, and decisions.
One person may perform more than one role for a small incident.
4. Contain the impact
Prioritize limiting further impact before attempting a permanent fix.
Possible containment actions include:
- Rolling back a recent deployment
- Disabling a failing feature
- Redirecting traffic
- Scaling a service
- Isolating an affected component
- Blocking suspicious requests
- Restoring service from a verified backup
Record each action and its result. Do not make irreversible changes without appropriate approval unless immediate action is required to protect users or data.
5. Investigate the cause
Use reliable evidence such as:
- Application logs
- Infrastructure metrics
- Traces
- Recent deployment history
- Configuration changes
- Database health indicators
- Security alerts
- User reports
Avoid changing multiple variables at the same time because this makes the root cause more difficult to identify.
6. Communicate regularly
Incident updates should be concise and factual. Include:
- Current impact
- Actions completed
- Current investigation
- Temporary workaround, if available
- Expected next update
- Risks or decisions requiring attention
Do not include credentials, personal data, or unverified conclusions in incident communications.
7. Resolve and verify
After applying a fix:
- Confirm that the affected service is operating normally.
- Check error rates, latency, resource usage, and key business metrics.
- Validate the result with representative user journeys.
- Continue monitoring for recurrence.
- Communicate that the incident is resolved only after verification.
8. Conduct a post-incident review
For significant incidents, complete a blameless review. Document:
- Incident summary
- Timeline of events
- Customer and business impact
- Detection method
- Root cause or contributing factors
- What went well
- What could be improved
- Corrective and preventive actions
- Owners and target dates
Actions should be specific, measurable, and tracked to completion.
Incident record template
- Incident title:
- Start time:
- End time:
- Incident owner:
- Affected systems:
- Severity:
- Impact:
- Timeline:
- Containment actions:
- Resolution:
- Root cause:
- Follow-up actions: