Rebalancing Pods with Descheduler
Kubernetes schedules pods when they're created. It picks the best node at that moment based on resources, affinity, taints, and topology. But it never revisits that decision. Over time, nodes get unbalanced. A node that was empty during a deploy gets overloaded after a few scaling events. Another node sits mostly idle because its pods got evicted and rescheduled elsewhere.
Descheduler fixes this. It runs periodically, evaluates the cluster state, and evicts pods that shouldn't be where they are. The regular scheduler then places them on better nodes.
How it runs
In my homelab, Descheduler runs as a Deployment (not a CronJob) with a 30-minute cycle. Every 30 minutes it evaluates the cluster and evicts pods that match any of the enabled policies. Running as a Deployment means it also exposes metrics continuously, which Prometheus scrapes via a ServiceMonitor.
It runs on the small tier node. Minimal resources: 50m CPU, 128Mi memory requested.
The policies
Seven plugins enabled, split into two categories:
Balance plugins redistribute pods across nodes:
LowNodeUtilization. If a node is using less than 30% CPU, memory, and pods, it's considered underutilized. If another node is above 60% on any of those, Descheduler evicts pods from the overloaded node so the scheduler can place them on the underutilized one. This keeps workloads spread more evenly.
RemoveDuplicates. If two replicas of the same Deployment end up on the same node (which can happen after a node restart or scaling event), one gets evicted so the scheduler can spread them.
RemovePodsViolatingTopologySpreadConstraint. If pods violate their topology spread constraints (e.g., should be spread across nodes but ended up on the same one), they get evicted and rescheduled properly.
Deschedule plugins fix pods that are in the wrong place:
RemovePodsHavingTooManyRestarts. Pods with more than 100 restarts (including init containers) get evicted. A fresh start on a potentially different node sometimes fixes issues caused by local state or node-specific problems.
RemovePodsViolatingNodeTaints. If a node gains a NoSchedule taint a running pod doesn't tolerate (can happen after a node is cordoned or relabeled), the pod gets evicted so the scheduler can move it somewhere it's allowed.
RemovePodsViolatingNodeAffinity. If a pod's requiredDuringSchedulingIgnoredDuringExecution affinity no longer matches the node it's on (can happen after node label changes), it gets evicted.
RemovePodsViolatingInterPodAntiAffinity. Pods that violate inter-pod anti-affinity rules get evicted so the scheduler can fix the placement.
Extra RBAC for PVCs
The default Descheduler RBAC doesn't include permissions to read PersistentVolumeClaims. Some eviction decisions need to check if a pod has local storage that would be lost. I added a ClusterRole and ClusterRoleBinding that gives the Descheduler read access to PVCs.
Observability
A ServiceMonitor scrapes Descheduler metrics into Prometheus, and a Grafana dashboard (loaded via ConfigMap) shows eviction counts, policy activity, and cycle duration. This is how I know the Descheduler is actually doing something useful and not just evicting pods for no reason.
