Deleting an environment: from two clusters to one

In Building a Homelab I explained why I ran two clusters on one mini PC. Develop to break things, production to keep them running. I said I'd split it the way most companies do.

I deleted develop this week. Six months was enough to show it was solving a problem I don't have.

Why develop stopped earning its keep

The argument for a develop cluster is that you need somewhere safe to break things. That holds when a team shares production and a bad chart change ruins someone else's afternoon. On one mini PC it gave me two identical copies of the same cluster, and I only ever used one of them.

It also assumes the other copy is precious. Mine isn't. A lab is supposed to be disposable: you destroy it and rebuild it from scratch on purpose. That's the old cattle, not pets line, or a phoenix server if you want a name for the single machine. If the whole thing is rebuildable in an afternoon, a second cluster protecting the first one is protecting nothing.

What develop actually cost me was a second cluster to maintain. Two sets of node VMs on the same host, two ArgoCD instances, two of every addon, holding RAM and disk that production had better uses for. And because I never really used it, it drifted. Charts got pinned at older versions there. Its Thanos storegateway was federated into production's query path, so a cluster I ignored was answering queries I cared about.

The tell was the values files. Every chart carried values.yaml, values-develop.yaml, and values-production.yaml, and the convention said all of them must exist even when identical. That's 38 files whose only job was to describe a difference that had stopped existing.

What removing an environment actually touches

More than the VMs. The environment label had leaked into five separate places, and they all had to move together:

terraform/production/cluster/   ─┐
terraform/production/bootstrap/  │  directory layout
                                 │
values-production.yaml (×38)     │  per-chart config
argo.prd.ruiz.sh                 │  DNS
kube context "production"        │  local kubeconfig
cluster="production"             │  Prometheus external label,
                                 ╵  tape's event, tailor's routing key

The last one is the interesting one. My right-sizing pipeline stamps a cluster label on every event, and the service that writes the recommendations back routes on it. Change the label without changing the router and the recommendations go nowhere, silently. No error, just a pipeline that stops having an effect.

I renamed the cluster to lab and collapsed everything into a single values.yaml per chart. Directories became terraform/cluster/ and terraform/bootstrap/. Hostnames dropped the env segment, so argo.prd.ruiz.sh is now argo.ruiz.sh, covered by one wildcard certificate.

The rebuild

Dropping develop freed the VM IDs that production wanted, so I rebuilt it with the env prefix gone from the node names too. Four nodes on Talos Linux v1.14.0, Kubernetes 1.37:

Same shape as before, one fewer copy of it. The tiers exist so nodeSelector means something: Prometheus and MinIO land on large, the operators sit on small.

What a cold rebuild exposes

This is the part worth writing down. I've applied this Terraform many times, but always against a cluster that already existed. Building from nothing found defects that had been there for months.

The network config never worked. The machine config patched interface = "eth0", and these VMs present their NIC as ens18. Talos wrote an address and a route for a link that doesn't exist, both inert, and every node quietly ran on whatever DHCP handed it:

patch says:   eth0    <node-ip>/22  via <gateway>   ╴╴> matches nothing
reality:      ens18   <node-ip>/22  via <gateway>   <── DHCP

It looked fine for as long as the router's reservations also supplied a gateway. On the rebuild two of them didn't, and those nodes came up with no route out, never reached an NTP server, and held their kubelet in Waiting for time sync forever. A time sync error that was really a routing error that was really a config that had never applied.

Selecting the NIC by MAC fixed it, and then broke DNS. Static addressing means no DHCP, and DHCP was where the nameserver had been coming from. The nodes fell back to Talos' built-in 1.1.1.1, CoreDNS forwards to the node's resolver, and suddenly nothing in the cluster could resolve my internal registry. Three things had been arriving from the router, and I'd only noticed two of them.

Renaming things that other things hold

Four times during this rebuild I hit the same shape of problem, and it took me until the fourth to name it.

The gateway. I renamed the Istio Gateway from homelab to lab. The new one sat pending forever because the old one still held the MetalLB IP, and ArgoCD wouldn't delete the old one because PruneLast=true defers pruning until the sync is healthy, and the sync couldn't be healthy without the IP.

A Loki pod. I changed the anti-affinity so a second replica could schedule. The running pod still carried the old hard rule, and the scheduler honors the rules of pods that already exist, so the new one stayed Pending behind a rule that only lived in a pod nobody had restarted.

Grafana CRs. Renamed objects reconciled fine; the ones that kept their name and only changed a selector couldn't, because instanceSelector is immutable. ArgoCD retried forever against an API that would never accept it.

Every one of these had correct config in git. The cluster just had no path to reach it on its own. Deleting the old object was the fix each time, and deleting it created no drift, because git already described the end state.

Making destroy and apply repeatable

The thing I actually wanted out of this was a rebuild I could run again without remembering anything. Most of that is fixed in the Terraform, but two parts can't be:

The bootstrap stage holds Helm releases living inside the cluster. Once the VMs are gone the provider can't reach them, so destroy fails on connection. Those objects die with the VMs anyway, so the state gets dropped instead of destroyed.

The cluster stage reads a talos_cluster_health data source, and Terraform re-reads data sources during the destroy refresh. Against a cluster that's already down, that blocks for minutes. -refresh=false skips it, and the Talos provider offers health only as a data source, never a resource, so there's nowhere in the config to express this.

Both live in a Makefile now, with the reasoning in the header rather than in my memory. make apply runs the cluster stage, exports the kubeconfig, waits for the apiserver to hold steady for 25 seconds, then bootstraps. That wait exists because the health check gates on etcd and node health, which says nothing about whether the control plane's static pods have stopped restarting. The first Helm install walked straight into that window and died.

I spent about six hours on this, and maybe forty minutes of it was the rename I set out to do. The rest was finding out which parts of my infrastructure had been working by accident. Every defect had the same shape: something wasn't declared, the environment supplied it, and nothing noticed until the environment changed.

The cluster is at 40 of 40 applications healthy now, on a Kubernetes version four minors newer than what it ran this morning. But the thing I'd keep from the day isn't the upgrade. It's that a rebuild from zero is the only honest test of whether infrastructure-as-code actually describes your infrastructure.