IT playbook · AI Employee: Ivy

Get Kubernetes Cluster Info from Datadog

Engineers get cluster snapshot in one turn

Install this playbookBack to marketplace
The problem

"What's the state of the payments cluster right now?" — an engineer asks in Slack while investigating an outage. The answer lives in Datadog + Kubernetes dashboards + Grafana + kubectl, requires stitching signals across all of them, and takes 10-20 minutes for someone who knows the tooling. During an incident, this delay compounds; nobody has one place to see the actual picture.

At a glance
Trigger
Chat
Approvals
None
What it does
Read-only
Systems
Datadog · Kubernetes API
How it feels in production

An hour-by-hour walkthrough.

SRE Priya asks Ivy: "what's the state of the payments-prod cluster?" Ivy queries Datadog + K8s API + Grafana in parallel and returns in 20 seconds: **Cluster: payments-prod (us-east-1)** - Version: 1.29.4, 12 nodes (all Ready) - Node types: 8 × m5.2xlarge (workload), 4 × m5.large (system) - CPU utilization: 62% (up from 45% 2h ago) - Memory utilization: 71% - Pod count: 342 across 18 namespaces, 340 Running, 2 Pending (see below) **Workload health** - Deployments: 47, all at desired replica count - Recent restarts: payments-api (3 restarts in last hour — investigate), notifications (1 restart) - HPA: 4 at max replicas (payments-api, checkout, session, webhook) **Pending pods** - payments-api-worker-abc123: PodPending, insufficient CPU on nodes - session-svc-def456: PodPending, insufficient memory - Recommendation: scale up node pool by 2 **Recent events** - 14:32: OOMKilled on payments-api pod (memory limit exceeded) - 14:28: Deploy of payments-api v2.4.1 completed **Correlated** - Datadog dashboard link: [payments-prod overview] - Grafana panels: [cluster overview] [payments-api detail] Ivy has done in 20 seconds what would take 15 minutes manually. Priya starts investigating from the actual state, not from partial views.
How it works

Step by step.

  1. 01

    Parse cluster + scope query

    Cluster identifier, scope (whole cluster, namespace, workload, node pool). Handles common shorthand (payments-prod, checkout-staging).

    Reasoning · Cluster registry
  2. 02

    Query multiple sources in parallel

    Datadog for metrics + logs, K8s API for state, Grafana for dashboards, kubectl-equivalent for events. All in parallel.

    Datadog · Kubernetes API · Grafana · Prometheus
  3. 03

    Correlate + synthesize

    State + metrics + events + recent changes correlated. Signals matched to common patterns (OOMs, restarts, pending pods, node pressure).

    Correlation engine · Pattern library
  4. 04

    Present with links + recommendations

    Structured summary + deep links to dashboards for drill-down. Recommendations for common issues.

    Slack · Teams · Deep-link generation
  5. 05

    Log queries + optimize common patterns

    Repeat query patterns become saved views. Common investigations get faster over time.

    Query log · Saved views
Systems and wiring

What you connect to make this run.

Datadog · New Relic · Grafana · Prometheus

read

Metrics + logs + APM data. Multi-source read for comprehensive view.

Kubernetes API · kubectl equivalent

read

Live cluster state: nodes, pods, deployments, events. Read-only for status queries.

PagerDuty · Incident tracker

read

Recent incidents affecting cluster; correlated with current state signals.

Slack · Teams

read+write

Primary interface for engineers. Query in-flow; answer in-flow.

What changes

Before and after, honestly.

Time to comprehensive cluster status
Before
10-20 minutes
After
Under 30 seconds
% of investigations starting from correct state
Before
50-70%
After
95%+
SRE hours per week on repeat status queries
Before
8-15 hours
After
1-3 hours
Time-to-diagnosis during incidents
Before
20-60 minutes
After
5-15 minutes
Related

Playbooks that pair with this one.

Security

Incident Response Orchestration

Cluster status is part of incident context.

IT

Outage Detection & Bulk Tagging

Cluster health signals feed outage detection.

IT

Network Troubleshooting

Cluster network status included in network troubleshooting.

Frequently asked

Answers about this playbook.

What about multi-cluster environments?

Multi-cluster queries supported: 'payments-prod + payments-staging' returns comparative view. Cluster-of-clusters (fleet view) for platform teams.

Can it make changes (scale, restart)?

Read-only by default. Change actions available via separate approve-then-execute flow with proper RBAC. Never silent writes.

How does it handle multi-cloud (AWS + GCP + Azure clusters)?

Provider-agnostic. Same query surface across EKS, GKE, AKS, self-hosted. Provider-specific details surfaced when relevant.

What about custom controllers or CRDs?

Custom resources supported when registered. Query the cluster for what exists; explain what's there. Custom pattern libraries added as needed.

How does it interact with GitOps (ArgoCD, Flux)?

GitOps state + drift surfaced. Cluster desired-state vs. actual-state comparison. Deployment intent visible alongside current state.

See it run on your data.

Free plan, no credit card. Connect the systems this playbook needs and run it against a past event first.

Book a demoMore IT playbooks