Profile Applicability:
 • Level 1

Description:
 The eventRecordQPS on the Kubelet configuration limits the rate at which events are gathered. Setting this parameter to an appropriate value ensures that security-relevant events are captured without overwhelming the Kubelet or event processing systems.

Rationale:
 It is essential to capture all relevant events to ensure consistent monitoring and quick detection of security incidents. However, setting this value too low may result in missing important events, while setting it to 0 may result in a denial of service due to excessive event generation.

Impact:
 Pros:

  • Ensures timely capture of important security events.

  • Provides security insights and analytics that are essential for incident detection.

Cons:

  • Setting this parameter too high might overwhelm event storage systems.

  • Setting it to 0 might lead to denial of service due to excessive event creation.

Default Value:
 The default value may vary depending on the Kubernetes distribution. Refer to AWS EKS documentation for default settings.

Pre-requisites:

  • SSH or Session Manager access to worker nodes.

  • Permissions to modify Kubelet configurations and restart services.

  • Kubernetes access to verify changes via /configz or deploy privileged pods.

Remediation

Test Plan:

Using AWS Console:

  1. SSH into each worker node and check the eventRecordQPS setting in the Kubelet service file:
    sudo grep "eventRecordQPS" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

  1. Review the value of the eventRecordQPS argument and determine if it is set appropriately for your cluster's event load.
    If the argument is not found in the service file, verify the Kubelet configuration file by checking the --config argument in the service file. Open the configuration file:
    sudo less /path/to/kubelet-config.json

  1. Verify that the eventRecordQPS is set to an appropriate level.

Using AWS CLI:

  1. Get the list of nodes in your cluster:

    kubectl get nodes

  1. Start a proxy to the Kubernetes API:

    kubectl proxy --port=8080

  1. In a new terminal, for each node, run:

    export NODE_NAME=my-node-name
curl -sSL "http://localhost:8080/api/v1/nodes/${NODE_NAME}/proxy/configz"


  1. Verify that the eventRecordQPS setting is configured appropriately in the returned JSON.

Implementation Plan

Using AWS Console:

  1. SSH into the worker node and locate the Kubelet configuration file.

  2. Open the file:
    sudo less /path/to/kubelet-config.json

  1. Set the eventRecordQPS to an appropriate value, such as 10 or 20, based on the event load of your cluster. For example:
    "eventRecordQPS": 10

  1. Save the file and ensure the Kubelet service file does not override this configuration.

  2. Reload systemd and restart the Kubelet service:
    systemctl daemon-reload
    systemctl restart kubelet.service
     systemctl status kubelet -l

Using AWS CLI:

  1. Edit the Kubelet configuration file and set eventRecordQPS to an appropriate value.

  2. If using systemd, ensure the service file (/etc/systemd/system/kubelet.service.d/10-kubeadm.conf) includes the eventRecordQPS argument, or add the following line:

    --event-record-qps=10

  1. Reload systemd and restart the Kubelet service:

    systemctl daemon-reload
 systemctl restart kubelet.service
systemctl status kubelet -l

Backout Plan

Using AWS Console:

  1. If issues arise, revert the eventRecordQPS setting to the previous value or disable it by setting it back to 0 or the default configuration.

  2. Restart the Kubelet service after rolling back the changes.

Using AWS CLI:

  1. Revert the changes made to the eventRecordQPS argument or configuration file.

  2. Restart the Kubelet service:

    systemctl daemon-reload
systemctl restart kubelet.service

References:

  1. Kubelet Documentation

  2. Reconfigure a Node's Kubelet

  3. Kubernetes GitHub - Kubelet Config