Tailor and tape: right-sizing Kubernetes workloads from real usage

Nobody knows what to put in resources the first time. You copy the block from another chart (a Helm package that defines how the app gets deployed), round it up because you'd rather waste memory than get paged, and ship it. Then it stays there. The workload gets rewritten, the traffic pattern changes, someone bumps the limit during an incident and never brings it back down, and two years later those numbers still describe a service that no longer exists.

I built two services to fix that. tape runs in each cluster and measures what containers actually use. tailor takes those numbers and opens a pull request against the chart repo with the requests and limits it thinks the container should have. Nothing is applied automatically, a human still reviews and merges the diff.

Why those names

A tailor doesn't guess your size. They take a measuring tape, write down the numbers, and only then cut the cloth. The measuring and the cutting are two different acts, done with two different tools, and the order matters. Off-the-rack resource requests are the copy-pasted 512Mi that fits nobody in particular.

So tape is the measuring tape. It goes around the workload, reads the numbers, and hands them over. It does no sizing at all. tailor is the one holding the scissors: it takes the measurements and cuts the values to fit that specific container on that specific cluster.

The pun works twice, because the tape is also what carries the measurements from the cluster to the service that uses them.

The shape of it

┌─ cluster A ─────────────────┐
│  tape ──PromQL──▶ Thanos    │──┐
└─────────────────────────────┘  │
┌─ cluster B ─────────────────┐  │ usage events
│  tape ──PromQL──▶ Thanos    │──┤
└─────────────────────────────┘  │
                                 ▼
                   ┌──────────────────────────┐
                   │ Kafka: rightsizing.usage │
                   └────────────┬─────────────┘
                                ▼
                          ┌──────────┐
                          │  tailor  │
                          └────┬─────┘
                               │ one PR per app
                               ▼
                        ┌──────────────┐
                        │  chart repo  │
                        └──────────────┘

One tape per cluster, one tailor for the whole fleet, a Kafka topic in between. Each tape queries only its own cluster's Thanos, so nothing is measured fleet-wide and no cross-cluster query has to fan out.

tape is a weekly CronJob, and each environment runs on a different day so the resulting pull requests don't all land in someone's inbox on a Monday morning. A run lists every Deployment, StatefulSet, DaemonSet and Argo Rollout in the cluster, keeps the ones whose pod template carries the opt-in annotation, resolves each one's chart repo from its ArgoCD Application, measures every container, and produces one event per (namespace, repo).

If a workload isn't annotated, tape never looks at it. If the repo can't be resolved from an ArgoCD Application, the workload is skipped rather than guessed at. Guessing which repo to open a PR against is the kind of mistake you only make once.

The split is a trust boundary

The two-service design isn't about microservices. It's about credentials.

   cluster read access                 git credential
          │                                  │
          ▼                                  ▼
    ┌──────────┐      usage event      ┌──────────┐
    │   tape   │ ─────────────────────▶│  tailor  │
    └──────────┘                       └──────────┘
   no git token                     no cluster access

Measuring needs read access to a cluster. Writing needs a token that can push branches and open PRs. Nothing in the system needs both, so nothing in the system has both. tape's RBAC is get and list on workloads and ArgoCD Applications, with no write verbs anywhere. It holds no secret, and the event it produces carries no credential.

tailor sits on the other side with a GitHub App instead of a PAT. For each event it resolves the app's installation on the target repo and mints a short-lived per-repo token, which authenticates the clone and the PR API and then dies with the throwaway checkout. That App is installed once at the org level and already covers every repo, so installing it isn't the opt-in step. Opting in is two smaller things: the annotation on the workload (o11y.ruiz.sh/tailor: "true", say) on the read side, and a tailor.yaml in the chart repo on the write side. No file there, no write, no matter what shows up on the topic.

What gets measured, and over how long

Everything comes from cadvisor and kube-state-metrics, scraped into the cluster's own Thanos. Six numbers per container, plus a count of how many distinct days of data exist.

Requests and limits deliberately look at different windows. A request should track steady state, so it gets the full two weeks. A CPU limit is about short bursts, and a two-week average flattens exactly the thing you're trying to size for, so the burst peak and the throttle ratio use a fixed 7-day window at a 5-minute step. Tuning the main window doesn't move them.

Two details in the PromQL took longer to get right than the policy did.

Collapse the pod label first. cadvisor series carry pod and id labels, so every pod restart starts a brand new series. Take a quantile straight off that and you get the p95 of one pod incarnation, which after a busy week of restarts can be laughably low. Every query wraps the selector in a subquery that does max by (namespace, container) first, then takes the quantile over that.

Scope the pod name tightly. The selector uses pod=~"^<workload>-[a-z0-9]+(-[a-z0-9]+)?$", which allows one or two kube-generated suffixes and nothing more. Without that anchor, measuring fluent-bit happily picks up fluent-bit-k8s-events too, and you size one workload from another's usage.

There's also a small thing about OOM kills that surprised me: container_oom_events_total never increments on cgroup v2 nodes. It's just dead. So the OOM signal comes from kube-state-metrics' last-terminated reason instead, the same series the OOM alerts already fire on.

Why Kafka and not a webhook

The first version had tape POST the event straight to a tailor webhook, which opened the PRs inline inside the request. That worked until GitHub's secondary rate limit tripped and handed back a Retry-After longer than the request budget. A whole cluster's work vanished until the next weekly run.

You can't wait out a rate limit inside a synchronous handler, so the fix was to stop trying. tape produces to a topic and forgets about it, tailor consumes at whatever pace GitHub allows, and the offset is only committed once the event is fully processed. A rate limit or a restart replays the event instead of losing it. The webhook is gone entirely, including for the on-demand path.

The event is keyed by cluster:namespace:repo, so a re-run supersedes the previous message for that group rather than piling up.

The write-side contract

Being measured is only half the opt-in. The other half is a tailor.yaml at the root of the chart repo, which is what tells tailor where in your values files a given container's resources block actually lives:

version: 1
file: chart/{env}/{app}/values-{cluster}.yaml
apps:
  metrics:
    prometheus-metrics:
      prometheus: monitoring.prometheus.configuration.resources
    metrics-tsg:
      store: monitoring.thanos.storeGateway.resources

The nesting is app, then workload, then container. The app is the grouping key, so one app is one PR. The workload has to be in the key because container names collide across workloads more often than you'd expect.

No tailor.yaml, no write. A malformed one is logged and skipped as a per-event error, because a broken file in one repo must never wedge the partition for everyone else.

The effective file path is required to contain {cluster}. That rule exists because the first thing someone does is point two clusters at the same values file, and then each run clobbers the other's sizing. Absolute paths and .. are rejected too, and the writer re-checks with realpath that the resolved path stayed inside the checkout.

The policy

For each container: memory request is p95, memory limit is peak plus 15%, CPU request is p95, CPU limit is the busiest burst times 1.5. Values snap to a ladder (128Mi, 256Mi, 512Mi, then 1Gi steps for memory) so nothing changes over a rounding error.

Three rules do the actual work, though.

A healthy CPU limit is only ever lowered. Hitting a CPU limit slows a container down, it doesn't kill it. If the limit could grow every time usage got close to it, you'd get a limit that ratchets upward forever. It grows only when the kernel reports real throttling, meaning a CFS throttled-periods ratio at or above 25%, and then it doubles each run until the throttling clears. That ratio catches something an average hides completely: a container can sit at 8% of its limit on average and still be clamped in half of its 100ms scheduling slices.

An OOM kill overrides the deadband. After a kill, the measured peak is worthless, because the kernel censored it at the very limit that did the killing. "Peak is comfortably below the limit" proves nothing when the limit is what stopped it. So a kill in the window re-derives the memory limit from peak plus buffer, ignores the 20% deadband, and never shrinks. It steps one ladder rung per run, since tailor genuinely cannot see how far above the ceiling real demand sits until it measures again at the higher ceiling.

A 20% deadband on everything else. A value only moves if the new target is more than 20% away from what's live. Week-to-week noise gets absorbed and produces no diff, and no diff means no PR at all.

Those guards compound into the property I actually care about: the policy is a fixed point. Once a value is applied, the next run on similar usage computes the same value and there's nothing to commit. Right-sizing becomes a one-time convergence per container instead of a weekly stream of pull requests nobody reviews.

The gap nobody thinks about

There's one failure mode I didn't anticipate. Request tracks p95 and limit tracks peak, which means a spiky workload drifts apart over time until you're looking at a 50Mi request under a 1Gi limit. Every value is individually correct. The pair is a problem, because the scheduler bin-packs by request, so the node has 50Mi reserved for something that will periodically take a gigabyte.

So there's a cap on the ratio between them. When the limit exceeds ten times the request, tailor raises the request rather than lowering the limit. Lowering the limit would give back the OOM protection and the burst headroom that the rest of the policy just worked to establish. Raising the request only costs you scheduling honesty.

A flat 10× turned out to be too loose at the top end, though. Ten times a 50m request is 450m of unreserved burst, which is noise. Ten times a one-core request is nine unreserved cores, which is a node going down. Past a threshold (one core, or 2Gi) a tighter 4× cap takes over. It keys off the request rather than the limit, so a small spiky workload keeps the generous cap and only genuinely large containers get squeezed.

Reviewing these PRs changed how I read a chart. Half the containers were sized for a load that never arrived, and a handful had been quietly OOM-killing for weeks under a limit somebody set once and stopped thinking about. Neither group would have shown up on a dashboard, because the whole point of an over-provisioned request is that nothing ever goes wrong with it. It just costs money forever.

Related reading: KEDA autoscaling handles the other axis of the same problem, how many replicas rather than how big each one is, and the descheduler deals with what happens when the scheduler's original placement decision goes stale.