How a sample moves through my VictoriaMetrics cluster

In October I switched off Prometheus and Thanos in the homelab. VictoriaMetrics is now the only place metrics live, and The Metrics Stack describes the setup it replaced.

The reason was one gap. With Thanos, a sample lived only on Prometheus' disk until the sidecar uploaded the closed two-hour block to the bucket, so losing that disk could cost up to two hours of metrics. VictoriaMetrics gets each sample out of vmagent within seconds and stores it on two nodes.

The whole path

The diagram below is live. Clay dots are samples on their way in, dark dots are queries on their way out, and the dashed lines are the operator handing out config. Click a pod to take it down and the traffic routes around it, or use the buttons to follow only the writes or only the reads.

The same path as text:

  ~113 targets ◄── GET /metrics ── vmagent ×3         ──┐
  agents, cluster ×2 ──── OTLP ──► gateway-metrics ×2 ──┤
                                   vmalert            ──┤      ┌── Grafana, vmui
                                                        │      ├── vmalert
                                                        ▼      ▼
                                              ┌───────────────────────────────┐
                                              │           vmauth ×2           │
                                              └─────┬───────────────────┬─────┘
                                                    ▼                   ▼
                                             ┌─────────────┐     ┌─────────────┐
                                             │ vminsert ×2 │     │ vmselect ×2 │
                                             └──────┬──────┘     └──────┬──────┘
                                   writes to 2 of 3 │                   │ all 3
                                                    ▼                   ▼
                                              ┌───────────────────────────────┐
                                              │  vmstorage ×3, one per worker │
                                              └───────────────────────────────┘

Three things write and two things read, and all of them go through vmauth.

Samples come in two ways. vmagent pulls them: it scrapes about 113 targets every 60 seconds, from the kubelet and the API server to every app that serves /metrics. The OpenTelemetry Collector pushes the rest: its agents and its cluster collector send OTLP to gateway-metrics, which turns it into Prometheus remote write. vmalert is the third writer, with the recording rules it computes from what is already stored.

Two charts deploy the VictoriaMetrics side, both in the victoria-metrics namespace. victoria-metrics-operator (sync wave 6) installs the operator and its CRDs, and victoria-metrics (wave 7) declares a VMCluster, a VMAgent and a VMAlert, all named lab. The operator turns those into workloads called <component>-lab, and it keeps vmagent's and vmalert's config in step with the scrape and rule resources in the cluster. Alerting stays in Grafana.

Nothing is scraped by a config file

Every target is a VictoriaMetrics resource that lives in the chart owning the thing being scraped. The VMAgent has selectAllByDefault: true, so the operator collects every scrape resource in the cluster and rewrites vmagent's config whenever one changes.

  VMServiceScrape / VMPodScrape / VMNodeScrape   (in the owning chart)
        │ watched by
        ▼
  victoria-metrics-operator ──writes──► scrape config ──reload──► vmagent ×3

Adding a scrape is a template in the chart that needs it, and nothing to edit in the metrics chart.

This is the one for kube-state-metrics:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  name: kube-state-metrics
  namespace: {{ .Release.Namespace }}
spec:
  jobLabel: app.kubernetes.io/name
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
      app.kubernetes.io/instance: {{ .Release.Name }}
  endpoints:
  - port: http
    honorLabels: true

jobLabel makes the series job="kube-state-metrics", which is what the kubernetes-mixin dashboards select on. honorLabels is there because these series describe other objects, so their own namespace and pod labels have to win over the scrape target's.

No chart renders a ServiceMonitor or a PrometheusRule anymore. Upstream charts keep their serviceMonitor option off and the wrapper chart declares a VMServiceScrape instead. The operator's Prometheus converter is off as well, so nothing needs the monitoring.coreos.com CRDs.

One target went missing on the way: the API server. vmagent rejects any scrape body over -promscrape.maxScrapeSize, 16MiB by default, and kube-apiserver serves about 23MiB, so VictoriaMetrics held no apiserver series at all. I raised the limit to 64MB. Setting it to 0, which means unlimited for Prometheus' body_size_limit, would make vmagent reject every scrape.

The push side

Some metrics only exist as OTLP: kubeletstats, k8s_cluster and whatever the apps instrument with the OpenTelemetry SDK. Those go through the Collector, and only gateway-metrics knows where VictoriaMetrics is:

  opentelemetry-agent (every node) ──┐
                                     ├── OTLP :4317 ──► gateway-metrics ×2
  opentelemetry-cluster ×2 ──────────┘                        │
                                          Prometheus remote write, WAL on disk
                                                              ▼
                                       vminsert-lab:8480 (vmauth ×2) ─► vminsert

The gateway runs two replicas as a StatefulSet. Each pod keeps the exporter's write-ahead log on its own 1Gi volume, so a restart resumes whatever vminsert hadn't acknowledged. The agents keep their own queue on the node's disk, so losing one gateway pod costs nothing either: they retry on the other one.

It writes Prometheus remote write rather than OTLP, even though vminsert accepts OTLP. The exporter translates OTLP the way OpenTelemetry's compatibility spec says, with service.name and service.instance.id becoming job and instance. vminsert's own OTLP ingestion doesn't do that: scraped series arrived as service_name, and up showed up as count:up1.

The gateway also drops a handful of resource attributes before writing. VictoriaMetrics turns every resource attribute into a label and drops a series past 40 labels. AKHQ's JVM metrics reached 39 and the OpenTelemetry demo's reached 42, and most of those labels (pod UID, start time, container ID, the process command line) change on every restart, so each restart also started a fresh set of series.

Recording rules

The kubernetes-mixin dashboards read precomputed series such as node_namespace_pod_container:container_memory_working_set_bytes. Those come from VMRules, and vmalert is the only thing that both reads and writes:

  VMRule ──► operator ──► vmalert ──┬── every 60s: query vmselect
                                    └── write the result to vminsert

It runs a single replica, since a second one would write every recorded sample twice. If it restarts, the recorded series pause and the next evaluation carries on from there. It has no notifier at all (-notifier.blackhole), because every alert rule lives in Grafana.

Four copies on purpose

vmagent runs in cluster mode: three members, one per worker, with a replication factor of two, so every target is scraped by two members at different instants. Each sample then goes to two of the three vmstorage, because the VMCluster also has replicationFactor: 2. That makes up to four records of the same point, and losing any single pod loses none of them.

Deduplication cleans up afterwards. vmstorage and vmselect keep one sample per series per minute, the scrape interval:

vmcluster:
  retentionPeriod: 180d
  replicationFactor: 2
  dedupMinScrapeInterval: 60s

vmagent:
  scrapeInterval: 60s
  extraArgs:
    promscrape.maxScrapeSize: 64MB
    promscrape.cluster.replicationFactor: "2"
    promscrape.noStaleMarkers: "true"
  members: 3
  queueStorageSize: 2Gi

That's a trimmed excerpt of charts/victoria-metrics/values.yaml. The queueStorageSize is a PVC per member where vmagent buffers samples while vminsert is unreachable, and replays them once it comes back.

Two things in this part bit me.

Staleness markers. A member that stops scraping a target writes a staleness marker for every series of it. Deduplication keeps the newest sample in each 60s window, so a marker that lands after the other member's real scrape throws the real sample away. I saw a stored sample disappear that way with nothing lost at the source. noStaleMarkers turns them off. The cost is that a target that goes away keeps its series visible for the query lookback instead of ending at once.

Shards restart together. The operator's shardCount makes every member its own StatefulSet and updates them all at the same time. On October 31 one change restarted all three members together and about a third of the targets lost a scrape. The members are now pods of a single StatefulSet, each taking its member number from its own pod name (promscrape.cluster.memberNum: "%{POD_NAME}"), and a rollout replaces one member at a time.

Why vmauth sits in front

The cluster docs ask for a load balancer that stops routing requests to unavailable nodes, and the plain Services fall short for both paths. vminsert-lab balances per TCP connection, so vmagent's long-lived connections stick to one pod. vmselect-lab is headless because vmselect is a StatefulSet, so a client resolves one pod IP and keeps it.

With requestsLoadBalancer on the VMCluster, the operator deploys vmauth and points both Services at it. It balances per request and skips unhealthy pods, and every URL that used those Services stays the same. It runs two replicas on two nodes, since a single balancer would be the one pod the rest of the replicas were meant to stop depending on.

Deleting pods on purpose

The buttons in the diagram come from a test I ran on October 11. I deleted one pod at a time while a probe read through vmauth about once a second, the same path Grafana uses, and waited for each pod to come back before the next one:

Q='/api/v1/namespaces/victoria-metrics/services/vmselect-lab:8481/proxy/select/0/prometheus/api/v1/query?query=count%28up%7Bjob%3D%22apiserver%22%7D%29'
while true; do
  r=$(kubectl --context lab get --raw "$Q")
  echo "$(date -u +%T) $(echo "$r" | jq -r '.status, (.isPartial // false)' | tr '\n' ' ')"
  sleep 1
done

Run it in a foreground terminal. As a background job, each round took about 30 seconds instead of one.

The probe can miss something between two reads, so I also watched vmagent's remote write errors and retries, vm_partial_results_total for queries answered without every vmstorage, and count_over_time(up{job="apiserver"}[1m]) for minutes with no sample.

vmagent was still a pair of pods back then. The three-member setup came on October 31 and hasn't been through the same test. I also haven't tested a whole node going down, two pieces at once, or a vmstorage gone for longer than a pod restart, which is why the diagram only lets you take down one pod at a time.

What it costs

Before the switch I ran both stacks side by side for a few days, scraping the same ~105 targets. These are six-hour averages from October 11:

VictoriaMetrics costs more here because everything runs twice: the scrapers, every stateless component and every sample. Per copy it lands in the same range as Thanos. The vendor benchmarks that promise several times less compare single copies.

Thanos also had two things I gave up. Its compactor downsampled old data into 5m and 1h tiers, and VictoriaMetrics OSS has no downsampling, so the 180 days of retention are raw samples and a 90-day graph reads all of them. And Thanos kept history in object storage, outside any worker. Here it sits on local disk on the nodes, so capacity grows by adding disk: the small worker went from 60 to 150GB to hold its share. Thanos' history wasn't migrated either.

There are no backups. Replication covers a dead disk, not a deletion.

Capacity is fine for now. The cluster docs say to keep half the RAM and CPU free under the real workload. On October 11, at ~263k active series, vmstorage peaked at ~29% of its memory limit, and read linearly it reaches the 50% line at around 450k. It's the first component to get there.

What I wanted from the switch was for one dead disk or one dead pod to cost nothing, and that part held up every time I deleted something. A whole node going down is the case I still owe it.

Sources

• VictoriaMetrics cluster version
• vmagent
• vmauth
• VictoriaMetrics operator
• VMServiceScrape