Service Health & Deployment Diagnostics

Objective: Automate deployment verification for {{service}} (can be all services) and send detailed summary reports to Slack.

Steps:

(1) Verify Service Health

• Execute a GET request using the HTTP Agent to:

{{service_url}}/healthy

Confirm that the service returns a healthy status.

(2) Fetch Recent Deployments (Github)

• Query for deployments of the {{service}} service using the GitHub Agent within the last 20 minutes.

(3) Analyze Resource Usage (Datadog)

• Check CPU and Memory usage of the deployed application using the Datadog Agent over the past 20 minutes. Ensure values are below 80% for both metrics.

(4) Kubernetes Resource Usage (Kubernetes)

• Retrieve memory and CPU usage of all pods in the {{service}} namespace using the Kubernetes Agent. Highlight any pods exceeding resource limits.

(5) Latency Check (Datadog)

• Run the following query to detect latency spikes in the last 20 minutes using the Datadog Agent:

trace.go.opentelemetry.io_contrib_instrumentation_github.com_gin_gonic_gin_otelgin.server{service:{{service}}}

Check for latency anomalies or spikes over the last 20 minutes.

---

(6) Error Rate Analysis (Datadog)

• Use this query to retrieve top error logs from the last 20 minutes using the Datadog Agent: status:error service:{{service}}

Monitor error rates and identify any recent spikes.

---

(7) Log Analysis (Datadog)

• Check all error logs with the following query: service:{{service}} status:error

Look for critical or recurring error messages.

---

(8) Slack Notification

• Send a message to Slack channel #on-call, tagging @team-oncall with the following format:

🚨 Runbook Executed: {{service}} Service Diagnostics

📅 Date & Time: {{execution_time}}
🔧 Service: {{service}}
🌐 Environment: {{environment_name}}
📁 Repository: {{service}}

Summary:
All diagnostic checks have been completed.

Detailed Steps:
-Health Check - ✅ Passed
- Deployment Activity - ✅ Found recent deployment
- Datadog Resource Usage - ✅ CPU/Memory below threshold
- Kubernetes Resource Usage - ✅ All pods within safe range
- Latency Metrics - ✅ No spikes
- Error Rate - ✅ Normal
- Logs - ✅ No recent critical errors

Result: ✅ SUCCESS

• If failures or anomalies are detected, adjust Result to:
  • ❌ FAILURE
  • ⚠️ CRITICAL

Add footer:

:alert: @team-oncall please review the runbook execution results.