Lestrade: AI triage for Grafana alerts

An alert fires at 3am. The on-call engineer wakes up, sees the chat notification, and now has to context-switch from sleep to investigation. Open the dashboard. Identify the cluster. Check the pod. Read the logs. Cross-reference the runbook, assuming there is one.

Lestrade is a small Python service that does the boring first pass for them. It catches the Grafana Alerting webhook, hands the alert off to an AI agent that has read-only access to the cluster and the observability stack, and posts the agent's analysis back to the same chat thread a few seconds later.

The name comes from Sherlock Holmes. Inspector Lestrade is the Scotland Yard liaison who brings the case from the crime scene to Holmes at Baker Street. Same job here: the alert is the crime scene, Lestrade is the inspector, the AI agent is the detective.

The idea didn't come from the 3am scenario, though. It came from watching Bret Fisher talk about how AI is changing DevOps careers. He spends a good chunk of it on observability, on where the repetitive human work still sits, and it stuck with me. The first ten minutes of triaging an alert are the same ten minutes every time, and none of them need me awake to do them. Lestrade is what I built after sitting with that thought.

How it sits

flowchart TD
    GA[Grafana Alerting] -->|"notification policy matches lestrade=true"| CP["contact point (webhook)"]
    CP -->|"POST /webhook (JSON)"| L[Lestrade]
    L -->|"GET /api/v1/provisioning/contact-points/export"| API["Grafana API<br>(discover chat URL)"]
    L -->|"POST A2A (JSON-RPC 2.0)"| AG["AI agent<br>(read-only cluster + observability MCP)"]
    AG -->|text response| L
    L -->|case report| GC[Google Chat space]

Three pieces matter here. The first is how Grafana decides to send the alert to Lestrade. The second is what Lestrade does with it. The third is the prompt that turns "here's a JSON payload" into "here's a root cause hypothesis".

Wiring it into Grafana

Two things to configure on the Grafana side.

Contact point. A webhook contact point pointing at the Lestrade service:

http://lestrade.svc:8000/webhook

That's where Grafana POSTs the alert payload.

Notification policy. At the top of the policy tree, a route that matches lestrade=true and sends to that contact point, with continue=true so the alert keeps flowing through the tree to whatever the team already has set up:

- match: { lestrade: "true" }
  receiver: lestrade-webhook
  continue: true
- match: { team: "platform" }
  receiver: platform-chat
- match: { team: "payments" }
  receiver: payments-chat

The continue: true is the important bit. Lestrade is additive enrichment, not a replacement. The team still gets the original notification through their existing path. They just also get the AI analysis a few seconds behind.

Required labels on the alert rule

Two labels on the alert rule wire the whole thing together:

The opt-in design matters. Not every alert benefits from AI triage. A clean SLO breach with a runbook does. A noisy CPU-utilization alert for a database that's been overprovisioned for two years does not. Teams pick which rules get the treatment.

What happens inside Lestrade

Grafana retries any webhook that doesn't respond within a few seconds. The AI analysis takes 20-60s. So the first thing the handler does on /webhook is return 200 immediately, then do the work in the background.

The dispatch is an asyncio.create_task bounded by two limits. A semaphore caps in-flight analyses (default 5) so a burst doesn't starve the event loop. A separate max_pending bound (default 100) caps how many tasks can sit queued behind that semaphore. As long as there's room, the handler returns 200 and runs the analysis in the background. Once 100 tasks are already pending, Lestrade stops accepting: it increments lestrade_webhook_overload_drops_total and returns 503. Grafana re-delivers on the next evaluation interval, by which time the backlog has usually drained.

Before dispatching, the handler applies two filters.

Opt-in re-check. The lestrade=true label is re-evaluated on each alert in the payload. The notification policy already filtered for this, but the re-check is defence-in-depth in case someone edits the policy tree.

Dedup. Grafana re-delivers the same alert group every evaluation interval (by default every minute) until the alert resolves. Lestrade suppresses those re-deliveries within a 1-hour TTL, so a single OOMKill alert produces one analysis, not 60 per hour.

The dedup key is more than just the groupKey, though. It's cluster|groupKey|status|startsAt. That startsAt is the subtle part. Grafana stamps a fresh startsAt every time an alert transitions resolved → firing. A flapping alert (fire → resolve → fire within the hour) would otherwise collapse into the first analysis and get silently swallowed, but each new firing is a new incident and deserves a fresh look. With startsAt in the key, sustained firing stays dedup'd (the same startsAt is re-delivered every eval), while a genuine flap re-analyses:

key = cluster | groupKey | status | startsAt

t0  firing    startsAt=T0    miss → analyse
t1  firing    startsAt=T0    hit  → dedup'd        (routine re-delivery)
t2  resolved
t3  firing    startsAt=T3    miss → analyse        ← flap: new startsAt, fresh look

The trade-off is that chronically flapping rules generate more analyses, which is intentional. Dedup is meant to suppress re-deliveries of the same occurrence, not to paper over a flapping bug; rules that flap chronically should be fixed (lower sensitivity, higher for:), not silenced by Lestrade.

The dedup cache is in-memory (LRU, bounded to 1000 entries by default). Pod restart drops both the cache and any in-flight analyses. Grafana re-delivers everything on the next eval, so nothing is permanently lost. Lestrade is stateless on purpose, no database, no Redis, no PVC.

A guard rail for runaway cardinality

One failure mode earns its own defence. A high-cardinality rule, say one built on a cross-cluster datasource with max by (pod, cluster), can expand to thousands of firing instances. Grafana groups them, and Lestrade renders every alert in a group into the agent prompt. A group carrying thousands of alerts becomes a giant, slow, expensive request that produces a generic answer at best and a context-window overflow at worst.

So there's a cap. Above WEBHOOK__MAX_ALERTS_PER_GROUP (default 20) accepted alerts in one group, Lestrade refuses to send the bloated prompt. The behaviour is configurable via WEBHOOK__ON_OVERSIZED_GROUP:

• reject (default): drop the whole group, spend zero tokens, and increment lestrade_alerts_filtered_total{reason="group_too_large"}. A group this big almost always means a rule that should narrow its group_by or alert on an aggregate, so the drop is a loud, actionable signal.
• truncate: keep the first N alerts, tell the agent how many were omitted, and still produce a representative analysis.

flowchart TD
    G[group of N accepted alerts] --> C{"N > 20 ?"}
    C -->|no| A[analyse the group]
    C -->|yes| M{WEBHOOK__ON_OVERSIZED_GROUP}
    M -->|reject| R["drop group · 0 tokens · group_too_large++"]
    M -->|truncate| T["analyse first 20 · prompt notes (N omitted)"]

The point isn't relevance. Every one of those alerts may be genuinely firing. It's that fifty pods crashing the same way are one diagnosis, not fifty. Twenty samples are plenty for the agent to spot the pattern; the fifty-first pod just burns tokens.

Discovering the chat URL from Grafana

Lestrade needs to POST the analysis to a Google Chat space. Each team has its own. The mapping from contact_point value to chat webhook URL lives in Grafana's contact-points config.

Rather than mirror that mapping into Lestrade's own config (and watch it drift), Lestrade queries Grafana directly:

GET /api/v1/provisioning/contact-points/export?decrypt=true

That returns a YAML export of every contact point with the actual webhook URL inline (Grafana decrypts secrets server-side when the API key has the right scope). Lestrade parses the export, builds a receiver_name → url dict in memory, and refreshes on-demand with a short cooldown when a team adds a new receiver.

The token for that API call is a Grafana service account token. The deployment pulls it from Vault at pod startup via the vault-secrets-webhook mutating admission pattern:

env:
  - name: GRAFANA__TOKEN
    value: "vault:secret/data/lestrade#GRAFANA_TOKEN"

That vault:… string gets resolved by the webhook before the env var lands in the container. The token never touches the manifest, the values file, or git.

Knowing who's actually using it

Discovering the chat URL taught Lestrade to talk to Grafana's provisioning API. The same channel answers a different question: which rules are opted in?

The firing metrics can't tell you. lestrade_alerts_received_total only ever counts rules that fired. An opted-in rule that's been quiet all week is invisible. To see real adoption you have to ask Grafana for the inventory.

So a background loop polls GET /api/v1/provisioning/alert-rules once an hour, keeps the rules that carry lestrade=true, and publishes them as a gauge:

lestrade_adopted_rule{alertname, datasource} 1

flowchart LR
    L[Lestrade] -->|"GET /api/v1/provisioning/alert-rules, hourly"| G[Grafana]
    G -->|"keep labels.lestrade == true<br>rebuild gauge"| M["lestrade_adopted_rule{alertname, datasource}"]
    M -->|scrape| P[Prometheus] --> D[dashboard]

One series per opted-in rule, rebuilt from scratch each cycle so an opt-out drops its series instead of going stale. count(lestrade_adopted_rule) is total adoption; count by (datasource) (...) shows how many rules ride the cross-cluster datasource versus a per-cluster one. The interval is ADOPTION__REFRESH_INTERVAL_SECONDS (default an hour, since rules change rarely and polling harder is wasted load), and it's fail-open: a failed poll keeps the previous gauge.

One thing it deliberately can't show is which clusters a rule covers. The cluster is a property of the data, not the rule. A cross-cluster rule never declares the clusters it spans. That answer stays with the firing metrics.

The preamble

When Lestrade calls the agent, it doesn't just forward the raw alert JSON. It composes a prompt with two parts: the alert context (cluster, namespace, pod, summary, runbook URL if any) and a fixed system instruction. The system instruction is the most opinionated part of the project, and it's what turns a generic LLM agent into something that produces consistent, on-call-friendly responses.

Highlights from the current preamble:

Scope. "Investigate only the specific alert described above. Do not investigate other unrelated alerts or pods you may find in the cluster." The agent has access to the whole cluster. Without this rule it wanders.

Tools. "Start with the Grafana MCP to understand the alert rule, its queries, and recent metric behaviour, then use other tools to investigate the cluster." This anchors the investigation in the rule itself before chasing symptoms.

Runbook handling. If the alert carries a runbook_url annotation, the agent fetches it and treats it as the team's documented procedure. If the fetch fails (404, auth, timeout), the agent doesn't retry forever. It adds a one-line note to the response so the team can fix the link, and continues the investigation as if the URL were absent.

Environment scope. "Scope all findings, evidence, impact, and recommendations strictly to the cluster and environment where the alert fired." Stops the classic LLM mistake of saying "in production" for a develop-cluster alert.

Output style. "Your response begins with *Alert:* on the first line. Nothing precedes it." No "Looking at the data, I can see…", no "After investigating…", no meta-narration. The on-call engineer reads the response in a chat thread and doesn't care about the agent's internal process.

Google Chat formatting quirks. Google Chat doesn't render markdown the way the LLM expects. So the preamble dictates: single asterisks for bold (*Root Cause:*), backticks around any technical string containing ~, _, or *, no ## headers, no ** double asterisks. Without this, half the response gets corrupted by Chat's formatter interpreting characters as italic or strikethrough.

The output template

The preamble ends with a fixed template the agent has to follow:

*Alert:* <alert name>
*Cluster:* <cluster name>
*Namespace:* <namespace>
*Component:* <component, container>
*Node:* <node info if relevant>

*Current State:*
<current metrics and status>

*Root Cause:*
<why the alert is firing>

*Evidence:*
<metrics, logs, or data that support the diagnosis>

*Impact:*
<what is affected and severity>

*Recommendation:*
<actionable steps to resolve>

Optionally, the agent can append a *Rule Suggestion:* section when the firing pattern indicates the rule itself needs tuning (flap, chronic firing, transient resolves under a few seconds). The preamble describes heuristics for when to include it, and forces concrete proposals: a specific numeric threshold, an exact for: duration, a label change. Vague suggestions get filtered out by the prompt itself.

The template does two things. It makes responses scannable, so the on-call engineer always knows where to look for the recommendation. And it forces the agent to commit. There's no *Root Cause:* section it can fill with "could be A or B or C" without consequence. It has to pick.

Lestrade appends a footer of its own to the agent's response before posting, separate from anything the model writes:

*Runbook:* <url>           (only if the alert carried one)
*Analysis time:* 34s

The *Analysis time:* line is the end-to-end latency of the agent call. It sets expectations in the thread (a 90s analysis reads differently from a 5s one) and gives a cheap, human-visible cross-check against the lestrade_kagent_latency_seconds histogram.

What you can see

Every stage above emits metrics, and there's a Grafana dashboard built on them. The ones worth knowing:

That last delivery metric, lestrade_gchat_no_url_total, is the canary for the one invariant the whole thing hinges on: the contact_point label on the alert must match, byte for byte, the name of a contact point that exists in Grafana. Get it wrong and the analysis is generated and then dropped, because Lestrade has nowhere to post it. The metric catches the typo before a human notices the silence.

What it isn't

Lestrade is read-only triage, not auto-remediation. The agent's tool set is limited to inspection: describe resources, get events, get logs, query PromQL and LogQL, fetch dashboards. No apply, no patch, no delete. Anything that needs doing about the alert stays a manual follow-up.

That's a feature, not a limitation. The AI hypothesis is a starting point for the engineer, not a decision. When the engineer sees *Root Cause:* OOMKilled, JVM heap exceeds the 256Mi limit, they still need to choose whether to bump the limit, tune the JVM, or fix the leak. Lestrade just gets them to that decision faster.

Lestrade dispatches over A2A (the agent-to-agent protocol, JSON-RPC under the hood) to an SRE coordinator, which is itself a kagent agent. A2A means Lestrade doesn't care what's on the other end, only that it speaks the protocol: today a kagent coordinator that wires up cluster tools, MCP servers for the observability stack, and the right model; tomorrow anything else that talks A2A. Without an agent already standing behind that endpoint, Lestrade would be a webhook with no brain to talk to.