Profile Applicability:
• Level 1
Description:
Enable kubelet client certificate rotation by ensuring the --rotate-certificates argument is either not present or set to true.
Rationale:
The --rotate-certificates setting allows the kubelet to rotate its client certificates automatically by creating new Certificate Signing Requests (CSRs) as its existing credentials expire. This automatic rotation prevents downtime due to expired certificates and ensures the availability of the cluster while maintaining the security properties of the CIA triad (Confidentiality, Integrity, and Availability).
Impact:
Pros:
Ensures continuous availability of the kubelet by preventing expired certificates from causing downtime.
Provides automated certificate rotation, reducing manual overhead.
Cons:
If using external certificate authorities (e.g., Vault), the rotation feature may not work and must be handled separately.
Default Value:
Refer to the Amazon EKS documentation for the default value, as it might vary by Kubernetes version and distribution.
Pre-requisites:
SSH or Session Manager access to worker nodes.
Permissions to modify Kubelet configuration files or systemd service arguments.
Kubernetes access to verify changes via /configz or deploy privileged pods.
Remediation
Test Plan:
Using AWS Console:
SSH into each worker node and run the following command to find the Kubelet process:
ps -ef | grep kubelet
If the --rotate-certificates argument is present, verify that it is set to true.
If the --rotate-certificates argument is not found, locate the Kubelet configuration file by checking the --config argument:
--config /etc/kubernetes/kubelet/kubelet-config.json
Open the Kubelet configuration file:
cat /etc/kubernetes/kubelet/kubelet-config.json
Verify that the RotateCertificate argument is either not present or set to true.
Using AWS CLI:
Get the list of nodes in your cluster:
kubectl get nodes
Start a proxy to the Kubernetes API:
kubectl proxy --port=8001
In a new terminal, for each node, run:
export NODE_NAME=my-node-name curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz"
Verify that the RotateCertificate setting is either not present or set to true in the returned JSON.
Implementation Plan
Using AWS Console:
SSH into the worker node and run ps -ef | grep kubelet to find the Kubelet process.
Open the Kubelet configuration file:
sudo less /path/to/kubelet-config.json
Modify or add the following entry to enable certificate rotation:
"RotateCertificate": true
Save the file.
Ensure that the Kubelet service file at /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf does not override this setting with --rotate-certificates=false.
Reload systemd and restart the Kubelet service:
systemctl daemon-reload
systemctl restart kubelet.service
systemctl status kubelet -l
Using AWS CLI:
Edit the Kubelet configuration file to set RotateCertificate to true.
If using systemd, add the following argument to the Kubelet service file:
--rotate-certificates=true
Reload systemd and restart the Kubelet service:
systemctl daemon-reload systemctl restart kubelet.service systemctl status kubelet -l
Backout Plan
Using AWS Console:
If issues arise, revert the Kubelet configuration file to its previous state without the RotateCertificate setting or set it to false.
Restart the Kubelet service after rolling back the changes.
Using AWS CLI:
Revert the changes made to the Kubelet configuration file or systemd service file.
Restart the Kubelet service:
systemctl daemon-reload systemctl restart kubelet.service