Security playbook · AI Employee: Sig

Incident Auto-Investigation via Grafana

Analyst reads summary in 2 min instead of grep across 5 systems

Install this playbookBack to marketplace
The problem

A P1 alert fires at 3am. On-call opens PagerDuty, opens Grafana, opens the SIEM, opens the CMDB to look up the service owner, opens Slack to find who deployed last, opens the audit log to look for privilege changes. Six tools before they start diagnosing. Every minute of that setup is a minute the incident isn't triaged. The Assistant should have done all six before waking anyone.

At a glance
Trigger
Monitor event (Grafana alert)
Approvals
None for read-only enrichment
What it does
Read-only
Systems
Grafana · Datadog · Okta · SIEM · Slack
How it feels in production

An hour-by-hour walkthrough.

3:14am. Grafana alert: latency spike on payments-api > 500ms sustained for 5 minutes. Webhook fires to Sig. 3:14am + 4 seconds. Sig has the alert. Pulls: the service (payments-api) and its owning team (payments-eng) from Context Graph; the last three deploys in the past 4 hours from your CI (a config change 42 minutes ago); recent identity events (nobody unusual accessed prod today); related SIEM signals in the past hour (nothing correlated). Drafts a summary: "Latency spike on payments-api. Correlated with a config deploy 42 min ago by @alice. No security signals. Suggested first action: consider rollback." 3:14am + 8 seconds. Sig posts the summary in the payments-incidents Slack channel, pages the on-call engineer via PagerDuty with the summary attached, opens an incident record in Incident.io with the full context bundled. 3:15am. On-call engineer opens Slack. Sees the summary, sees the deploy correlation, doesn't spend 10 minutes hunting. Rolls back. Latency recovers. Post-incident: the config change is fixed; the postmortem writes itself from the timeline Sig already built. When Sig detects a signal it can't correlate — an alert with no service ownership, an alert with no recent changes — the escalation still fires with the null-result summary so the human knows they're starting from scratch. No pretending.
How it works

Step by step.

  1. 01

    Ingest alert payload

    Webhook from Grafana / Datadog / Splunk / Prometheus. Sig parses alert name, service, severity, threshold breached, time.

    Grafana · Datadog · Splunk · Prometheus
  2. 02

    Enrich with user, device, service context

    From Context Graph: which service owns the alert, which team owns the service, who's on-call, recent identity events for accounts with access to this service.

    Context Graph
  3. 03

    Correlate with recent identity + deploy events

    Pull the last N deploys from your CI (GitHub Actions / CircleCI / Vercel). Check for privilege changes in the past hour from your identity provider. Check SIEM for related signals in the same time window.

    GitHub · CircleCI · Okta · SIEM
  4. 04

    Draft summary + candidate root cause

    The Assistant drafts a summary of what happened, what changed recently, and a candidate root cause based on correlation strength. If nothing correlates, says so explicitly.

    LLM · Knowledge Studio (runbooks)
  5. 05

    Post to incident channel

    Post the summary in the service's incident channel, page on-call via PagerDuty / Incident.io with the summary attached, open the incident record with the full timeline pre-populated.

    Slack · PagerDuty · Incident.io
Systems and wiring

What you connect to make this run.

Grafana · Datadog · Splunk · Prometheus

trigger

Alert-manager webhook configured for high-severity alerts. Signed payload verified; Sig enqueues the investigation.

Context Graph

read

Read service-owner + team + on-call + identity context. Everything Sig knows about the involved parties.

GitHub · CircleCI · Vercel · Netlify

read

Read recent deploys / merges. API credentials with read-only workflow scopes.

SIEM (Splunk · Sumo Logic · Datadog Cloud SIEM)

read

Query related security events in the alert time window. Read-only search API.

PagerDuty · Incident.io

write

Page on-call with the summary. Open incident record with pre-populated timeline. Never resolves incidents autonomously.

What changes

Before and after, honestly.

Time from alert to on-call has context
Before
10-30 minutes across tool-hopping
After
10-30 seconds
Incidents traced to a specific change
Before
40-60% (rest attributed to "unknown")
After
75-90% (correlation exposes causes that would have stayed hidden)
MTTR on config-change-related incidents
Before
45-90 minutes
After
10-20 minutes (rollback candidate identified immediately)
Postmortem time
Before
2-4 hours reconstructing timeline
After
30-60 minutes; timeline already built
Related

Playbooks that pair with this one.

Security

Security Alert Triage & Context

The general-case enrichment playbook that runs on all alerts, not just Grafana.

Security

Incident Response Orchestration

The full-incident-management playbook that runs when a P1 escalates.

Security

Anomalous Login

The related pattern for auth-based alerts.

Security

Compromised Credential Response

Fires when the alert correlates with a credential incident.

Frequently asked

Answers about this playbook.

Does Sig ever take remediation actions autonomously?

No. Sig investigates and drafts summaries; humans decide on remediation. Auto-remediation is a separate opt-in playbook per specific pattern (e.g. auto-restart a container that OOM-killed, if you've approved that specific auto-response). The default is: investigate, notify, wait.

What about alerts with no correlation?

Sig says so explicitly: "No recent deploys, no privilege changes, no related SIEM signals in the window." The null result is itself useful — on-call knows they're not looking for a change, they're looking for a new failure mode.

Can we tune what Sig considers 'correlated'?

Yes. Per-service correlation rules — how far back to look, which change events matter, which SIEM signals are relevant. Defaults tuned for most services; you override where it matters.

How does this coexist with our existing runbook?

Sig indexes your runbooks via Knowledge Studio. On the alert investigation, Sig cites the relevant runbook section ("Runbook for high-latency incidents: check config-change-first") so on-call has the human-authored playbook + the machine-drafted investigation side by side.

What if we don't use Grafana?

Same pattern works for Datadog, Splunk, New Relic, PagerDuty (as trigger source), custom webhooks. The playbook is investigation-source-agnostic.

See it run on your data.

Free plan, no credit card. Connect the systems this playbook needs and run it against a past event first.

Book a demoMore Security playbooks