Profile Applicability:
• Level 1
Description:
Enable Kubelet server certificate rotation to ensure that the Kubelet's server certificates are periodically rotated, preventing potential downtime due to expired certificates.
Rationale:
The --rotate-certificates setting ensures that the Kubelet automatically requests a new serving certificate after bootstrapping its client credentials. It also rotates certificates as they near expiration. This feature ensures the continuous availability of the Kubelet by preventing downtime due to expired certificates and helps maintain the security of the cluster by ensuring up-to-date certificates.
Impact:
Pros:
Prevents downtime from expired certificates.
Automates the certificate management process, reducing manual intervention.
Improves security by ensuring the Kubelet is always using valid certificates.
Cons:
If Kubelet certificates are provided by an external source (e.g., Vault), you will need to manage rotation separately.
Default Value:
Refer to the Amazon EKS documentation for the default value, which may vary depending on Kubernetes version and distribution.
Pre-requisites:
SSH or Session Manager access to worker nodes.
Permissions to modify Kubelet configuration files or systemd service arguments.
Kubernetes access to verify changes via /configz or deploy privileged pods.
Remediation
Test Plan:
Using AWS Console:
SSH into each worker node and run the following command to find the Kubelet process:
ps -ef | grep kubelet
If the --rotate-kubelet-server-certificate argument is present, verify that it is set to true.
If the argument is not found, locate the Kubelet configuration file using --config and open it:
cat /etc/kubernetes/kubelet/kubelet-config.json
Verify that RotateKubeletServerCertificate is either not present or set to true under the featureGates section.
Using AWS CLI:
Get the list of nodes in your cluster:
kubectl get nodes
Start a proxy to the Kubernetes API:
kubectl proxy --port=8001
In a new terminal, for each node, run:
export NODE_NAME=my-node-name curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz"
Check the returned JSON to verify that RotateKubeletServerCertificate is set to true under the featureGates section.
Implementation Plan
Using AWS Console:
SSH into each worker node and find the Kubelet configuration file using ps -ef | grep kubelet.
Open the file:
sudo less /path/to/kubelet-config.json
Add or modify the following entry under the featureGates section to enable certificate rotation:
"featureGates": {
"RotateKubeletServerCertificate": true
}
Save the file.
Ensure that the Kubelet service file at /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf does not override this setting with --rotate-kubelet-server-certificate=false.
Reload systemd and restart the Kubelet service:
systemctl daemon-reload
systemctl restart kubelet.service
systemctl status kubelet -l
Using AWS CLI:
Edit the Kubelet configuration file and set RotateKubeletServerCertificate to true under the featureGates section.
If using systemd, add the following argument to the Kubelet service file:
--rotate-kubelet-server-certificate=true
Reload systemd and restart the Kubelet service:
systemctl daemon-reload systemctl restart kubelet.service systemctl status kubelet -l
Backout Plan
Using AWS Console:
If issues arise, revert the RotateKubeletServerCertificate setting to the previous state or set it to false.
Restart the Kubelet service after rolling back the changes.
Using AWS CLI:
Revert the changes made to the RotateKubeletServerCertificate argument or configuration file.
Restart the Kubelet service:
systemctl daemon-reload systemctl restart kubelet.service