Profile Applicability:
 • Level 1

Description:
Ensure that audit logs are collected and managed in accordance with the enterprise’s audit log management process across all Kubernetes components.

Rationale:
 Audit logs provide visibility into the activities occurring within a Kubernetes cluster, enabling the detection and investigation of security incidents and policy violations. Proper collection and management of audit logs are essential for maintaining an audit trail and ensuring compliance with security policies.

Impact:
 Pros:

  • Enables traceability of activities within the Kubernetes cluster.

  • Supports incident response and forensic investigations.

  • Helps meet regulatory and compliance requirements.

Cons:

  • May require additional storage and compute resources.

  • Misconfiguration or improper management of logs can pose security risks.

  • Requires consistent log forwarding and retention practices.

Default Value:
 By default, Kubernetes does not enable detailed audit logging. Configuration is required to enable and manage audit logs.

Pre-requisites:

  • Access to the Kubernetes cluster with kubectl.

  • IAM permissions for managing worker node policies, ConfigMaps, and audit-related pods.

  • A centralized logging destination like Amazon CloudWatch, Elasticsearch, or a log aggregation pipeline.

  • jq command-line JSON processor installed (for parsing output).

Remediation

Test Plan:

Using AWS Console:

  1. Navigate to the Amazon EKS > Clusters section in the AWS Management Console.

  2. Select your cluster, go to the Nodes tab.

  3. Use AWS Systems Manager Session Manager or EC2 Connect to log into a worker node

  4. Review the kubelet audit policy by inspecting its configuration.

  5. Validate that audit logs are being forwarded to a central location (e.g., CloudWatch Logs, Fluent Bit, or external logging stack).

Using AWS CLI:

  1. Get kubelet audit policy from the node.

  2. Check if logs are being collected.

  3. Verify integration with centralized logging.

kubectl get --raw /api/v1/nodes/${NODE_NAME}/proxy/configz | jq '.kubeletConfig.auditPolicy'
kubectl get --raw /api/v1/nodes/${NODE_NAME}/proxy/stats/summary | jq '.auditLogs'

Implementation Plan

Using AWS Console:

  1. Create an audit-policy.yaml file locally with the desired rules.

  2. Open the EKS node group EC2 instances via Session Manager or EC2 Connect.

  3. Place the policy under /etc/kubernetes/ or equivalent path used by the distribution.

  4. Modify the kubelet startup configuration to include --audit-policy-file and --audit-log-path options.

  5. Restart the kubelet process or the node.

  6. Set up log shipping to CloudWatch Logs using Fluent Bit, Fluentd, or a custom forwarder.

Using AWS CLI:

  1. Create the audit policy file.

  2. Apply the policy to the cluster.

  3. Forward logs using a custom pod.

apiVersion: audit.k8s.io/v1
kind: Policy
 rules:

  • level: Metadata
    resources:

    • group: ""
       resources: ["pods"]

kubectl apply -f <path-to-audit-policy>.yaml

kubectl create configmap cluster-audit-policy --from-file=audit-policy.yaml -n kube-system
kubectl apply -f - <<EOF
 apiVersion: v1
 kind: Pod
 metadata:
 name: audit-logging
 namespace: kube-system
 spec:
 containers:
 - name: audit-log-forwarder
 image: my-log-forwarder-image
 volumeMounts:
 - mountPath: /etc/kubernetes/audit
 name: audit-config
 volumes:
 - name: audit-config
 configMap:
 name: cluster-audit-policy
EOF

Backout Plan

Using AWS Console:

  1. Navigate to your cluster nodes via EC2 Console or Session Manager.

  2. Remove the audit policy file or revert kubelet configuration changes.

  3. Restart the kubelet service.

  4. Delete any ConfigMaps or DaemonSets created for log forwarding.

Using AWS CLI:

  1. Delete the audit policy ConfigMap and log forwarder pod.

  2. Revert the audit policy from the cluster.

kubectl delete configmap cluster-audit-policy -n kube-system
kubectl delete pod audit-logging -n kube-system
kubectl delete -f <path-to-audit-policy>.yaml

References:

  1. Kubernetes Audit Logging Documentation

  2. Audit Policy Configuration