Description
This control checks that node-pool upgrades on Utho Kubernetes are performed without dropping cluster capacity, by provisioning replacement nodes before draining the old ones (the effect a 'max surge' setting provides on other platforms).
Utho node pools do not expose a max-surge parameter, and recycling drains and replaces nodes in place. To achieve zero-downtime replacements you emulate surge behaviour: either temporarily raise the pool's Node Count so spare capacity exists before draining, or add a new node pool on the target version and migrate workloads onto it before removing the old pool. PodDisruptionBudgets protect critical workloads during the drain.
It passes when the upgrade process demonstrably adds capacity before draining and protects workloads with PDBs; it fails when nodes are drained in place with no surge capacity.
In-place node recycling that drains before replacements are Ready, dropping capacity mid-maintenance.
Critical workloads with no PodDisruptionBudget guarding minimum availability during drains.
Upgrade windows where Ready node count dips below the level needed to run all pods.
Rationale
Upgrading or recycling a node pool in place drains existing nodes before their replacements are ready, so cluster capacity dips mid-maintenance. Under load this forces pod evictions and downtime, and an attacker probing during the shrunken window finds a degraded, less-resilient service.
Provisioning surge capacity first keeps total Ready capacity flat across the upgrade, so pods reschedule onto new nodes without a gap. PodDisruptionBudgets make this safe by preventing a drain from taking a workload below its minimum available replicas.
Keeps Ready capacity flat so pods reschedule without an availability gap.
PodDisruptionBudgets stop a drain from breaching minimum replica counts.
Removes the degraded maintenance window an attacker could exploit.
Impact
Pros:
Node replacements happen with no capacity dip or user-visible downtime.
Critical workloads keep their minimum replica count throughout the upgrade.
Upgrades can run during business hours instead of a risky off-hours window.
Cons:
Temporary extra nodes are billed for the duration of the upgrade window.
Needs enough account/region capacity to run old + new nodes simultaneously.
Slightly more orchestration (scale up, cordon, drain, scale down) than a plain in-place recycle.
Default Value
Utho node pools have no max-surge setting; recycling drains and replaces nodes in place without first provisioning surge capacity, so any headroom during an upgrade must be added by you.
Pre-Requisite
Utho console/API access to change node counts and add node pools.
kubectl access to cordon, drain, and watch node/pod state.
PodDisruptionBudgets defined for critical workloads.
Enough account/region capacity to run temporary extra nodes.
Remediation
Test Plan
Using the Utho Console:
Review the node-pool upgrade runbook for this cluster.
Confirm the process raises the pool's Node Count (or adds a new pool on the target version) so replacement capacity is Ready before any drain.
Run 'kubectl get pdb -A' and confirm critical workloads have PodDisruptionBudgets.
During a test upgrade, watch 'kubectl get nodes' and pod availability and confirm Ready node count does not dip below the level needed to run all pods.
Pass: the upgrade adds capacity before draining and PDBs protect critical workloads, with no capacity dip observed.
Fail: nodes are recycled/drained in place with no surge capacity, or critical workloads have no PDB.
Using the Utho API:
The Utho API can confirm the pool's node count over the upgrade window, but the surge behaviour and PDBs are verified with kubectl, not via the API.
curl -s -H 'Authorization: Bearer $UTHO_API_TOKEN' https://api.utho.com/v2/kubernetes/<cluster_id>
Pass if node count is raised (or a new pool added) before the old nodes are removed, as seen across successive API reads plus 'kubectl get pdb -A' showing budgets on critical workloads; Fail if the count only drops during a recycle with no prior increase.
Implementation Steps
Using the Utho Console:
Before upgrading, scale the pool up by raising Node Count in the Utho console (or add a new node pool on the target version) so temporary replacement capacity is Ready first.
Ensure critical workloads have PodDisruptionBudgets so 'kubectl drain' cannot evict below the minimum available replicas.
Cordon and drain the old nodes only after the new nodes report Ready and pods have rescheduled onto them.
Remove the old nodes/pool, then return the pool to its normal Node Count.
Watch Ready node count and pod availability throughout to confirm no capacity dip occurred.
Using the Utho API:
Utho has no surge/upgrade-strategy API; emulate surge with the pool APIs below plus in-cluster kubectl cordon/drain.
curl -s -X POST -H 'Authorization: Bearer $UTHO_API_TOKEN' -H 'Content-Type: application/json' https://api.utho.com/v2/kubernetes/<cluster_id>/nodepool/add -d '{"nodepools":[{"label":"pool-surge","size":"<size_id>","count":"2","ebs":[{"disk":"30","type":"nvme"}]}]}'
Backout Plan:
If the new nodes misbehave, uncordon the original nodes (they are still Ready) so workloads keep running, and do not drain them.
Delete the temporary surge pool/nodes to return to the pre-upgrade layout and cost.
Confirm with 'kubectl get nodes' and pod availability that the cluster is back to its original healthy state.