Autoscaling with KEDA

Kubernetes has built-in HPA (Horizontal Pod Autoscaler), but it only scales on CPU and memory. The component I most wanted to scale, the ArgoCD repo-server, has a different bottleneck: it spends its time cloning and fetching Git repositories during reconciliation. The signal that matters is the Git request rate, which is a Prometheus metric, not CPU. That's where KEDA comes in.

What KEDA does

KEDA (Kubernetes Event-driven Autoscaling) extends the HPA with custom triggers. Instead of just CPU and memory, you can scale on Prometheus queries, queue depth, cron schedules, HTTP traffic, and dozens of other sources. Under the hood it still creates a regular HPA; KEDA just feeds it the external metrics.

The repo-server is the right target for this. When several Applications reconcile at once, or someone hard-refreshes a handful of apps, the repo-server does a burst of Git fetches and becomes the slow part of a sync. The rest of the time it sits mostly idle.

The ScaledObject

One ScaledObject, on the argocd-repo-server Deployment, with two triggers:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: argocd-repo-server
  namespace: argocd
spec:
  scaleTargetRef:
    kind: Deployment
    name: argocd-repo-server
  pollingInterval: 30
  cooldownPeriod: 300
  minReplicaCount: 1
  maxReplicaCount: 2
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 300
  triggers:
    - type: cpu
      metricType: Utilization
      metadata:
        value: "70"
    - type: prometheus
      metadata:
        serverAddress: http://thanos-query-frontend.metrics.svc.cluster.local:9090
        threshold: "10"
        query: sum(rate(argocd_git_request_total{namespace="argocd"}[2m]))

It scales from 1 to 2 replicas. The Prometheus trigger is the interesting one: it watches the rate of Git requests over a 2-minute window and scales up past 10 req/s. The CPU trigger at 70% utilization is a fallback for the cases where the repo-server is busy but not because of Git.

KEDA reads the metric through Thanos Query Frontend rather than Prometheus directly, so the same query path the dashboards use also drives autoscaling.

Tuning the threshold

The threshold started at 5 req/s, and that was a mistake. Baseline reconcile load alone, with around 15 Applications polling every 3 minutes, already sits at roughly 5 to 6 req/s. So the trigger was permanently crossed: both replicas stayed up around the clock, which defeats the whole point of scaling on spikes.

Raising it to 10 req/s leaves headroom above the steady-state churn. Now the second replica only comes up when something actually bursts (a wave of syncs, a manual refresh across many apps), and the cluster idles back to a single replica the rest of the time.

This is the kind of number you can't guess up front. You set it, watch the replica count over a few days, and find the line that sits above normal noise.

How the metric gets there

The repo-server doesn't expose argocd_git_request_total by default. The chart turns on its metrics endpoint and a ServiceMonitor so kube-prometheus-stack discovers and scrapes it:

repoServer:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: kube-prometheus-stack

From there the path is:

flowchart TD
    RS["repo-server emits argocd_git_request_total"] -->|ServiceMonitor scrape| P[Prometheus]
    P --> QF[Thanos Query Frontend]
    QF -->|"sum(rate(...[2m])), polled every 30s"| KA[KEDA metrics adapter]
    KA -->|"git rate > 10 req/s OR CPU > 70%"| HPA["HPA scales argocd-repo-server 1 ↔ 2"]

Scale up is quick (a 30-second stabilization window), scale down is deliberately slow (a 5-minute cooldownPeriod plus a 300-second stabilization window). A short dip in traffic shouldn't kill the extra replica only to recreate it a minute later.

KEDA config

KEDA runs on the small tier node. Three deployments: the operator, the metrics adapter (which bridges KEDA metrics to the Kubernetes external-metrics API), and the admission webhooks for validation.

All three have ServiceMonitors enabled, so Prometheus scrapes KEDA's own metrics. No CPU limits (only memory limits), following the same pattern as the rest of the cluster. The ScaledObject itself is opt-in per environment and enabled on both develop and production.

Why not just HPA

Plain HPA on CPU would miss the actual bottleneck. A repo-server can be slow on Git I/O (waiting on remote fetches, unpacking refs) while its CPU looks unremarkable, so CPU-based scaling reacts late or not at all. Scaling on the request rate that causes the load is more direct: when reconciles pile up, the Git rate climbs, and that's exactly when a second replica helps clear the queue.