GitOps with ArgoCD

Building the homelab ended with Terraform installing ArgoCD and pointing it at the repo. Everything running on the cluster is managed from there. This is how that part is wired.

I wanted a workflow where the cluster could rebuild itself from Git: destroy it, run Terraform, and ArgoCD reconciles everything else back into place. The repo is the source of truth, not the cluster.

The structure

Two pieces. The wrapper Helm charts that describe each component, and a single app-of-apps chart that turns a list of those components into ArgoCD Applications.

charts/                      # one wrapper chart per component
├── cert-manager/
│   ├── Chart.yaml           # declares the upstream chart as a dependency
│   ├── values.yaml          # baseline
│   ├── values-develop.yaml  # env override
│   ├── values-production.yaml
│   └── templates/           # optional extras (ServiceMonitors, dashboards)
└── ...

app-of-apps/                 # generates one Application per chart
├── Chart.yaml
├── values.yaml              # the list of charts, grouped by sync-wave
├── values-develop.yaml      # environment: develop
├── values-production.yaml   # environment: production
└── templates/
    └── application.yaml     # the generator

Each wrapper declares the upstream chart as a dependency instead of vendoring it, and overrides through values.yaml. The per-env values-develop.yaml and values-production.yaml merge on top, and both clusters render from the exact same charts.

One list, one Application each

The whole inventory of what a cluster runs lives in app-of-apps/values.yaml, grouped by sync-wave. Reading it top to bottom is the boot order:

charts:
  - wave: 2
    apps:
      - { name: cert-manager, namespace: cert-manager }
      - { name: istio-gateway, namespace: istio-system }
      - { name: minio, namespace: minio }
  - wave: 6
    apps:
      - { name: kube-prometheus-stack, namespace: metrics }
      - { name: loki, namespace: logging }
  - wave: 7
    apps:
      - { name: thanos, namespace: metrics }

The chart's one template ranges over that list and emits an Application per entry, pointing at charts/<name> and loading the right env values file:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  project: default
  source:
    repoURL: https://github.com/getulioruiz/homelab
    targetRevision: main
    path: charts/cert-manager
    helm:
      valueFiles:
        - values-production.yaml
      ignoreMissingValueFiles: true
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - ApplyOutOfSyncOnly=true
      - PruneLast=true

ignoreMissingValueFiles is what lets a chart skip having a per-env file. prune + selfHeal means a manual change in the cluster gets reverted on the next reconcile. Every generated Application also carries a retry/backoff policy, so a chart that fails to sync because its dependency isn't ready yet just retries instead of going Degraded and staying there.

Why app-of-apps and not an ApplicationSet

An ApplicationSet is the obvious tool for "generate N Applications from a list", and I tried it first. The problem is ordering. The ApplicationSet controller creates all the generated Applications at once, and the sync-wave annotation only orders resources within a single Application, not across separately-created ones. So istiod could come up before istio-base had laid down its CRDs, and the sync would fail.

The app-of-apps pattern sidesteps that. A single root Application owns all the generated children, so ArgoCD treats their sync-wave annotations as one ordered set and brings them up wave by wave. Same "one list" ergonomics, but with the boot order actually respected.

Sync waves

Some components depend on others. The Doppler operator needs to exist before doppler-secrets can sync. Istio's istiod needs istio-base for its CRDs. Sync waves let ArgoCD honor that order, same in both envs:

Waves -2 to 2 are the hard dependencies, where getting the order wrong means a chart fails to render. Above that it's mostly about making sure storage and secrets exist before the things that use them.

Bootstrap

ArgoCD can't install itself, and MetalLB has to hand out LoadBalancer IPs before ArgoCD's server can claim one. Both come from the Terraform bootstrap, covered in the homelab build.

The only thing Terraform hands to the cluster after that is one root Application pointing at app-of-apps/:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-100"
spec:
  source:
    repoURL: https://github.com/getulioruiz/homelab
    targetRevision: main
    path: app-of-apps
    helm:
      valueFiles:
        - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

Wave -100 makes the root settle before anything it generates. From there ArgoCD takes over: it renders the child Applications and reconciles them in wave order, including ArgoCD managing itself.

Adding a chart

The payoff is how little it takes to add something. Create charts/<name>/ with a Chart.yaml, a values.yaml, and any per-env overrides. Then add one line under the right wave in app-of-apps/values.yaml:

  - wave: 6
    apps:
      - { name: <name>, namespace: <ns> }

That's it. No per-cluster Application file to write, no generator config. Both clusters pick it up from the same list, each loading its own values-<env>.yaml. To remove it, delete the line.

The biggest win is confidence. I can destroy a cluster and rebuild it from scratch, and the repo brings it all back in the right order. The biggest lesson was that "generate Applications from a list" and "bring them up in order" are two different problems, and the boring app-of-apps chart solves both with less magic than the tool built for the job.