Generate Incident Postmortems Automatically

TL;DR

This runbook streamlines post-incident reviews by gathering alert data, relevant logs, and code diffs to generate a first-draft postmortem document and post it in Slack.

Who is this for?

SREs, platform engineers, and tech leads who want faster, more consistent postmortem documentation following production incidents.

What problem does this solve?

After incidents, engineers often forget to capture key context — or spend hours gathering it from multiple tools. This automation saves time and improves accountability.

Solves:

• Inconsistent postmortems
• Manual digging through logs and commits
• Poor knowledge sharing after incidents

What this workflow accomplishes

• Listens for Datadog incidents with severity “critical”
• Queries Splunk logs from the incident time window
• Fetches GitHub commits and PRs tagged with the incident ID
• Generates a Google Doc or Markdown summary
• Posts a link and summary to the #incidents Slack channel

Integrations

This runbook uses the following integrations:

• Datadog Agent: Detects new incidents and extracts tags and timestamps.
• Splunk Agent: Pulls recent logs tied to the incident scope.
• GitHub Agent: Queries PRs and commits matching the incident tag or affected service.
• Slack Agent: Notifies the team with the generated report and preview.

Setup

• Datadog:

  • Valid API and App Keys
  • Incident integration enabled
  • Datadog Agent installed

• Splunk:

  • API token with access to logs
  • Logs tagged with service, env, etc.
  • Splunk Agent installed

• GitHub:

  • OAuth or PAT with repo scope
  • Commit messages or PRs reference incident IDs (e.g. INC-123)
  • GitHub Agent installed

• Slack:

  • Bot token with chat:write permissions
  • Slack Agent installed

Runbook Template