Profile Applicability:
 • Level 1

Description:
 Do not disable timeouts on streaming connections by setting the --streaming-connection-idle-timeout argument to 0. Ensure that appropriate idle timeouts are configured for security and resource management.

Rationale:
 Setting idle timeouts ensures that inactive connections are automatically closed, protecting the system against Denial-of-Service attacks and reducing the potential for running out of ephemeral ports. By default, --streaming-connection-idle-timeout is set to 4 hours, but this might be too high for some environments. Adjusting this to an appropriate value ensures idle connections are timed out properly.

Impact:
 Pros:

  • Helps mitigate DoS attacks by ensuring inactive connections are closed automatically.

  • Reduces risk of resource exhaustion and ephemeral port depletion.

  • Enforces better resource management and security practices.

Cons:

  • Long-lived connections may be interrupted if they are idle for too long.

Default Value:
 By default, --streaming-connection-idle-timeout is set to 4h0m0s. This might be too high for your environment, and adjusting this to a lower value could enhance security.

Pre-requisites:

  • SSH or Session Manager access to the worker nodes.

  • Permissions to modify Kubelet configuration files or service arguments.

  • Kubernetes access to verify changes.

Remediation

Test Plan:

Using AWS Console:

  1. SSH into each worker node and run the following command to find the Kubelet process:
    ps -ef | grep kubelet

  1. Look for the --streaming-connection-idle-timeout argument in the Kubelet command. If it’s set to 0, this needs to be adjusted.

  2. If the --streaming-connection-idle-timeout argument is not present, locate the configuration file using --config, and open it:
    cat /etc/kubernetes/kubelet/kubelet-config.json

  1. Verify that the streamingConnectionIdleTimeout is not set to 0.

Using AWS CLI:

  1. Get the list of nodes in your cluster:

    kubectl get nodes

  1. Start a proxy to the Kubernetes API:

    kubectl proxy --port=8001

  1. In a new terminal, for each node, run:

    export NODE_NAME=my-node-name
curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz"

  1. Verify that the streamingConnectionIdleTimeout is set correctly and not to 0 in the returned JSON.

Implementation Plan

Using AWS Console:

  1. SSH into each worker node and find the Kubelet configuration file via ps -ef | grep kubelet.

  2. Open the configuration file:
    sudo less /path/to/kubelet-config.json

  1. Add or modify the following entry to set the streamingConnectionIdleTimeout to a non-zero value, such as 4h0m0s:
    "streamingConnectionIdleTimeout": "4h0m0s"

  1. Save the file.
    If using systemd (for Amazon Linux or Bottlerocket AMIs), edit the Kubelet service file (/etc/systemd/system/kubelet.service.d/10-kubelet-args.conf) and ensure the following argument is set:

  1. Reload systemd and restart the Kubelet service:

Using AWS CLI:

  1. Edit the Kubelet configuration file to set streamingConnectionIdleTimeout to 4h0m0s.

  2. If using systemd, ensure the --streaming-connection-idle-timeout argument is correctly added to the service file as follows:

    --streaming-connection-idle-timeout=4h0m0s

  1. Reload systemd and restart the Kubelet service:

    systemctl daemon-reload
systemctl restart kubelet.service
systemctl status kubelet -l

Backout Plan

Using AWS Console:

  1. If issues arise after disabling the --streaming-connection-idle-timeout setting, revert the changes in the Kubelet configuration or systemd service file.

  2. Restart the Kubelet service.

Using AWS CLI:

  1. Revert the Kubelet configuration file to its previous state if necessary.

  2. Remove or adjust the --streaming-connection-idle-timeout argument and restart the Kubelet service:

    systemctl daemon-reload
systemctl restart kubelet.service

References:

  1. Kubelet Documentation

  2. Kubernetes Pull Request on Streaming Timeout