Alerting to Slack

Having metrics and logs is only useful if something tells you when things go wrong. In the homelab, alerts land in a #alerts channel in Slack.

The first version of this had two paths: Alertmanager routing Prometheus and Loki rules, and Grafana Unified Alerting for rules created in the UI. Two routing trees, two message templates, two places to check when silencing something. I consolidated everything on Grafana Alerting and turned Alertmanager off.

One path instead of two

flowchart LR
    T[Thanos] -->|"PromQL, every 1m"| R[Grafana-managed rules]
    R --> P[Notification policy]
    P --> C[Slack contact point]
    C --> S["#alerts"]

Every alert now follows the same route: Grafana evaluates the rules against Thanos, the notification policy groups and throttles, the contact point posts to Slack.

The deciding constraint was Grafana's built-in Alertmanager: it only handles Grafana-managed rules, it can't ingest alerts pushed from an external Prometheus. So keeping any rules on the Prometheus side meant keeping a standalone Alertmanager running just to route them. Moving the rules into Grafana removed that reason to exist, and two lines in the kube-prometheus-stack values finished the job:

alertmanager:
  enabled: false
defaultRules:
  create: false

The second line drops the upstream defaultRules pack, dozens of generic rules that mostly produced noise on a four-node cluster. They're replaced by ten curated rules, more on those below.

Provisioned, not clicked

The weak point of the old Grafana path was that alerts were created in the UI, which means they lived in a database instead of Git. The fix is grafana-operator: it reconciles Grafana* CRs against the existing kube-prometheus-stack Grafana over its admin API. The Grafana instance itself stays owned by the kube-prometheus-stack chart; the operator only provisions the alerting resources into it.

flowchart LR
    G[Git] --> A[Argo CD] --> CR["Grafana* CRs"] --> O[grafana-operator]
    O -->|admin API| GF[Grafana]

Four CRs describe the whole alerting setup, all synced by Argo CD like everything else:

• GrafanaContactPoint. The Slack receiver. The webhook URL is injected from a Secret via valuesFrom, so it never appears in the CR.
• GrafanaNotificationTemplate. The message layout: status, severity, cluster, namespace, summary, description, and a runbook link when one exists.
• GrafanaNotificationPolicy. Groups by alertname, severity, and namespace. Default repeat is 12 hours, critical repeats every hour, warning every 6. This mirrors the old Alertmanager route on purpose: the behavior didn't change, only the machinery.
• GrafanaAlertRuleGroup. The rules themselves, evaluated every minute into a Homelab folder.

The rules

Ten rules, all PromQL against Thanos. Evaluating against Thanos instead of Prometheus means one rule set covers every cluster the query layer sees, and the cluster label comes along into the Slack message.

One thing Grafana-managed rules cost you: each rule is a three-step pipeline (an instant query, a reduce, a threshold expression) instead of a single PromQL expr with a comparison. The YAML is noticeably more ceremony than a PrometheusRule. It's the price of rules that Grafana itself evaluates, and it's worth paying once instead of running a whole extra component.

What happened to log alerts

The Loki Ruler used to fire log-based alerts at Alertmanager. That path left the cluster with it. A log-based alert is now just another Grafana-managed rule, querying the Loki datasource instead of Thanos; the current pack happens to be all PromQL. The Ruler's remaining job is recording rules that turn log patterns into Prometheus metrics, which the log stack post covers.

Secrets

flowchart LR
    D[Doppler] --> DS[DopplerSecret] --> K[Secret slack-credentials]
    K -->|valuesFrom| CP[GrafanaContactPoint]

The Slack webhook URL never touches Git. It lives in Doppler, gets synced into the metrics namespace by the Doppler Kubernetes Operator, and the contact point references it with valuesFrom. No environment variable injection, no provisioning-file tricks, and rotating the webhook is an update in Doppler followed by the operator resyncing.

Consolidating didn't change what pages me or when; the notification policy replicates the old Alertmanager routing exactly. What it removed was a component to run, a second template language to maintain, and the split brain of asking "which system owns this alert?" every time something fired. Ten curated rules in Git also turned out to be worth more than a hundred upstream defaults I never tuned.