The Metrics Stack

The metrics foundation in the homelab is kube-prometheus-stack, a Helm chart that installs Prometheus, Grafana, Node Exporter, and kube-state-metrics in one go. On top of it, a separate Bitnami Thanos chart handles long-term storage in MinIO.

The values described here are the production configuration.

What's in the stack

What gets scraped

flowchart TD
    P[Prometheus] -->|scrape every 60s| NE["Node Exporter<br>(DaemonSet, per node)"]
    P --> KSM["kube-state-metrics<br>(cluster state)"]
    P --> SM["ServiceMonitors per addon<br>(Thanos, ArgoCD, MinIO, Istio…)"]
    P --> OA["o11y-apps PodMonitor<br>(label opt-in)"]

Prometheus scrapes everything on a 60-second interval:

Node Exporter runs as a DaemonSet on every node. CPU, memory, disk, network, filesystem. The basics that tell you if a node is healthy.

kube-state-metrics exposes the state of Kubernetes objects. Pod status, deployment replicas, PVC capacity, node conditions. This is where you get metrics like "how many pods are in CrashLoopBackOff" or "is this deployment fully rolled out."

ServiceMonitors from each addon. Thanos, ArgoCD, MinIO, cert-manager, Istio. Each chart creates its own ServiceMonitor, so Prometheus discovers and scrapes them automatically.

PodMonitor for applications. A generic PodMonitor called o11y-apps watches for pods with the o11y.ruiz.sh/metrics: "true" label. It scrapes ports named metrics or http-metrics. This is how app pods opt into metrics collection without needing their own ServiceMonitor.

Prometheus and the WAL

Prometheus runs on the large tier node with a 30 GB PVC on local-path storage. Key settings:

• Retention: 4 days or 20 GB, whichever comes first
• Scrape interval: 60s
• WAL compression: enabled

Every metric goes to the WAL first, before anything else. The WAL is an append-only log on disk that acts as a buffer between incoming data and the final TSDB blocks.

flowchart LR
    S[scrape] --> W["WAL<br>(append-only)"] --> H["head block<br>(in memory)"] --> B["2h block<br>(on disk)"]

If Prometheus crashes, the WAL is what lets it recover without losing data. On startup, Prometheus replays the WAL to reconstruct the head block. Without it, any data scraped since the last 2-hour block would be gone.

The Thanos Sidecar only uploads completed 2-hour blocks. Data in the WAL and head block hasn't been shipped to MinIO yet, which means there's always a window of up to 2 hours of data that only exists locally.

Thanos: why and how

Prometheus keeps 4 days locally. For a homelab where I sometimes want to compare "this week vs the same week last month", 4 days isn't enough.

The Sidecar runs inside the Prometheus pod, watches the local TSDB directory, and uploads completed 2-hour blocks to MinIO. From there, the Store Gateway serves them back to Thanos Query, which deduplicates results from the Sidecar (recent) and Store Gateway (historical) into a single Prometheus-compatible endpoint.

flowchart LR
    P[Prometheus] --> BL["Block (2h)"] --> SC[Sidecar] --> M[(MinIO)] --> SG[Storegateway]
    SG -->|historical| Q[Thanos Query]
    SC -->|recent| Q
    Q --> G[Grafana]

The Query Frontend sits in front of Query as a caching layer. It splits large time-range queries into smaller chunks and caches the results in Memcached. Repeated queries return from cache.

The Compactor is the background worker. It merges small 2-hour blocks into larger ones (fewer blocks means less I/O on MinIO and faster queries) and downsamples old data:

A query for "CPU usage last 6 months" doesn't need per-second granularity. 1-hour resolution is enough and uses far less storage.

What I don't run:

• Ruler is disabled. Alerting is Grafana-managed rules, so there's nothing for a Thanos Ruler to evaluate.
• Receive is disabled. The homelab uses sidecar mode (Prometheus ships to MinIO via the Sidecar). Receive is for push-based ingestion, which adds complexity without benefit here.

Multi-cluster federation

The homelab runs two clusters, develop and production. Each has its own kube-prometheus-stack, its own Thanos, and its own MinIO bucket.

Production's Thanos Query federates develop's Store Gateway over the LAN. Develop pins its Store Gateway to a fixed MetalLB IP on port 10901, and production references it in query.stores. The result is a single Grafana that queries both clusters, with deduplication keyed on the prometheus_replica and cluster external labels.

Each cluster owns its MinIO bucket and runs its own Compactor. Two Compactors writing to the same bucket would corrupt each other.

Grafana

Grafana runs on the medium tier node. Datasources:

• Thanos Query for everything (default)
• Prometheus for the most recent data, direct
• Loki for logs, so the same Grafana serves the log stack too

Grafana's default datasource is Thanos Query, not Prometheus directly. The same dashboards work for both recent and historical data without picking the right datasource.

Dashboards load automatically via a sidecar that watches for ConfigMaps with the grafana_dashboard: "1" label across all namespaces. Each addon can ship its own dashboard as a ConfigMap.

Admin credentials come from Doppler via the Doppler operator. Grafana has a 5 GB PVC for persistence.

Alerting

There's no Alertmanager. The chart bundles one, but it's disabled: alerting runs as Grafana-managed rules provisioned by grafana-operator, evaluated against Thanos, and delivered straight to Slack. The upstream defaultRules pack is off too, replaced by a small curated set.

See alerting to Slack for the whole design and the rules.

Node placement

Prometheus, the Compactor, and the Storegateway sit on large because they're either memory-hungry or block-heavy. Querying components stay on medium, where the load is bursty but bounded.

The 60-second scrape interval is a tradeoff. It keeps storage and CPU low, but you lose granularity for short-lived spikes. For a homelab it's fine. In production I'd use 15 or 30 seconds for critical services.

The Compactor is the component that surprised me most. It runs quietly in the background, but without it, the bucket fills up with thousands of tiny 2-hour blocks and queries get slow. Compaction and downsampling are what make long-term storage practical.