Profile Applicability

  • Level 2

Description

Enhanced Monitoring for Amazon MSK brokers provides additional insights into the operations of your Kafka clusters by collecting and analyzing an expanded set of metrics in Amazon CloudWatch. This feature helps diagnose potential issues, optimize performance, and ensure the reliability of your Kafka clusters.

Rationale

  • Improved Observability: Provides detailed metrics for better visibility into Kafka broker performance.

  • Proactive Issue Resolution: Helps identify and resolve potential bottlenecks or failures before they impact operations.

  • Compliance: Aligns with best practices for monitoring and maintaining critical infrastructure.

Impact

Pros:

  • Enables detailed monitoring of Kafka brokers.

  • Facilitates proactive troubleshooting and performance optimization.

  • Enhances operational efficiency and reliability.

Cons:

  • May incur additional costs for CloudWatch metrics storage.

  • Requires configuration changes to enable enhanced monitoring.

Default Value

By default, MSK clusters use Basic Monitoring. Enhanced Monitoring must be explicitly enabled.

Pre-Requisite

IAM Permissions:

  • kafka:ListClusters

  • kafka:DescribeCluster

  • kafka:UpdateMonitoring

  • AWS CLI installed and configured.

Remediation

Test Plan:

 Using AWS Console:

  1. Sign in to the AWS Management Console.

  2. Navigate to the MSK Dashboard.

  3. Select Clusters from the left-hand menu.

  4. Click on the cluster name to view its details.

  5. Check the Monitoring Metrics section under the Properties tab. Verify that the monitoring level is set to Enhanced.

Using AWS CLI:

  1. List All MSK Clusters:

    aws kafka list-clusters --query "ClusterInfoList[*].{ClusterName:ClusterName,MonitoringLevel:Monitoring.MetricsLevel}"

  2. Verify the monitoring level for each cluster.

Implementation Steps:

 Using AWS Console:

  1. Sign in to the AWS Management Console.

  2. Navigate to the MSK Dashboard.

  3. Select the cluster to update.

  4. Click Actions and choose Edit Monitoring.

  5. Select Enhanced Monitoring and save the changes.

Using AWS CLI:

  1. Enable Enhanced Monitoring:

    aws kafka update-monitoring --cluster-arn <cluster-arn> --enhanced-monitoring PER_BROKER

  2. Verify the Changes:

    aws kafka describe-cluster --cluster-arn <cluster-arn> --query "ClusterInfo.Monitoring.MetricsLevel"

Backout Plan

Using AWS Console:

  1. If enabling Enhanced Monitoring causes issues or increases costs unexpectedly, sign in to the AWS Management Console.

  2. Navigate to Amazon MSK, select the Kafka cluster, and go to Edit.

  3. Disable Enhanced Monitoring by selecting Basic Monitoring or the desired monitoring level.

  4. Save the changes and ensure the Kafka cluster is no longer using Enhanced Monitoring.

Using AWS CLI:

  1. To revert to basic monitoring, run the following command:

    aws kafka update-cluster-configuration --cluster-arn <CLUSTER_ARN> --enhanced-monitoring "BASIC" --region <REGION>

  2. Verify that Enhanced Monitoring has been disabled by describing the cluster again:

    aws kafka describe-cluster --cluster-arn <CLUSTER_ARN> --region <REGION>

References

CIS Controls

Version

Control ID

Control Description

IG1

IG2

IG3

v8

5.3

Securely Manage Network Infrastructure – Ensure network devices are resilient and fault-tolerant.

●

●

●

v8

13.2

Ensure Secure Network Communication – Implement measures that prevent disruption during network changes.

●

●

●