Description
This control checks that each production Utho Kubernetes node pool keeps at least two worker nodes, so no pool can be reduced to a single node that is a single point of failure.
Because Utho pools are defined by a fixed Node Count (and, where autoscaling is layered on with the open-source Cluster Autoscaler, by a --nodes=<min>:<max> range), the floor is set both by the pool's base count in the console/API and by the autoscaler's minimum. The control passes when both the base count and any autoscaler minimum are 2 or more for every production pool.
It fails when a production pool can run on, or scale down to, a single node.
Production node pools with a base Node Count of 1.
Autoscaler --nodes ranges whose minimum is 1, allowing scale-down to a single node.
Workloads with no anti-affinity that could still land all replicas on one node.
Rationale
A node pool that can drop to a single node has no fault tolerance. One node reboot, a hardware failure, or a drain during maintenance takes the last replica offline, and an attacker who can crash or overload that one node causes a full service outage.
Keeping a floor of at least two nodes means routine maintenance and single-node failures degrade rather than down the service, and it gives Kubernetes somewhere to reschedule pods immediately instead of waiting for a replacement node to be provisioned.
Survives single-node reboot, failure, or maintenance drain without full downtime.
Gives the scheduler a live target to reschedule evicted pods onto.
Combined with pod anti-affinity, prevents all replicas from co-locating on one node.
Impact
Pros:
Removes the single-node single point of failure for production workloads.
Allows node maintenance/recycling with no total-outage window.
Improves resilience to volumetric abuse aimed at one node.
Cons:
At least two nodes run at all times, raising baseline infrastructure cost.
Small/dev workloads may not need the extra node, so a blanket rule can over-spend there.
True zone-level redundancy also depends on region support for spreading nodes.
Default Value
Utho does not enforce a minimum node count; a pool can be created with a single node (count = 1), so multi-node fault tolerance is opt-in and must be set explicitly.
Pre-Requisite
Utho console access to the cluster and its node pools.
An Utho API token for scripted pool creation/inspection.
kubectl access to the cluster to confirm Ready node count.
A list of which pools are production and require the two-node floor.
Remediation
Test Plan
Using the Utho Console:
In the Utho console open Kubernetes and select the cluster.
Under Manage Node Worker Pools, read the Node Count for each production pool.
If the Cluster Autoscaler is deployed, download the kubeconfig and check the autoscaler's --nodes=<min>:<max> args, confirming min is 2 or greater for each production pool.
Run 'kubectl get nodes' and confirm at least two Ready worker nodes exist per production pool.
Pass: every production pool has a base Node Count >= 2 and any autoscaler minimum is >= 2.
Fail: any production pool has a Node Count of 1 or an autoscaler minimum of 1.
Using the Utho API:
Read the cluster's node pools and their counts via the Utho API; the autoscaler minimum still has to be confirmed with kubectl.
curl -s -H 'Authorization: Bearer $UTHO_API_TOKEN' https://api.utho.com/v2/kubernetes/<cluster_id>
Pass if every production node pool in the response reports count >= 2 (and the autoscaler minimum, checked via kubectl, is also >= 2); Fail if any production pool reports count = 1.
Implementation Steps
Using the Utho Console:
In the Utho console open Kubernetes, select the cluster, and under Manage Node Worker Pools raise each production pool's Node Count to at least 2.
If the Cluster Autoscaler is in use, edit its deployment so the pool's --nodes minimum is 2 or greater, so scale-down never drops below two nodes.
Where the region supports it, spread the pool across nodes/zones and add pod anti-affinity so replicas do not co-locate on one node.
Verify with 'kubectl get nodes' that at least two Ready worker nodes exist per production pool.
Using the Utho API:
Create or extend pools with a count of 2 or more via the Utho API; the autoscaler minimum is still set in-cluster with kubectl.
curl -s -X POST -H 'Authorization: Bearer $UTHO_API_TOKEN' -H 'Content-Type: application/json' https://api.utho.com/v2/kubernetes/<cluster_id>/nodepool/add -d '{"nodepools":[{"label":"prod-pool","size":"<size_id>","count":"2","ebs":[{"disk":"30","type":"nvme"}]}]}'
Backout Plan:
If the extra node causes cost or capacity issues on a non-critical pool, lower its Node Count in the console, but keep production pools at >= 2.
If an autoscaler minimum change misbehaves, revert the --nodes minimum to its previous value and re-apply the autoscaler deployment.
Confirm with 'kubectl get nodes' that the pool still reports the intended Ready node count after any revert.