Profile Applicability:
- Level 1
Description:
This check ensures that the Amazon OpenSearch Service domain is configured with at least three dedicated master nodes. Dedicated master nodes are responsible for managing cluster-wide actions, such as creating and deleting indices, managing shards, and maintaining cluster metadata. Having at least three master nodes provides fault tolerance and ensures high availability for the cluster's control plane.
Rationale:
Having at least three dedicated master nodes ensures that the OpenSearch cluster remains highly available even if one or two master nodes fail. This configuration helps prevent potential single points of failure and ensures that the cluster can continue to manage critical operations, even under failure scenarios. A minimum of three master nodes also ensures that the cluster can maintain quorum in case of node failures, thereby preventing split-brain situations, where two master nodes could assume control, leading to potential data corruption.
Impact:
By configuring at least three dedicated master nodes, you ensure that your OpenSearch cluster maintains high availability and fault tolerance for critical cluster operations. This setup minimizes the risk of service disruption and ensures that cluster management operations remain available even during node failures. However, adding more master nodes might incur additional cost and resource usage.
Default Value:
By default, OpenSearch Service domains are not configured with dedicated master nodes. If not configured, OpenSearch may use data nodes for managing cluster-wide operations, which can negatively affect performance and scalability.
Pre-Requisites:
AWS account with sufficient permissions to manage OpenSearch domains.
Knowledge of the desired OpenSearch cluster configuration.
Familiarity with OpenSearch architecture and node roles.
Remediation:
Test Plan:
Using AWS Console:
Log in to the AWS Management Console.
Navigate to Amazon OpenSearch Service.
Choose the Domains option from the left pane.
Select the OpenSearch domain to check.
In the Cluster Configuration section, verify the number of dedicated master nodes:
Ensure that Dedicated Master Nodes is enabled.
Verify that the Dedicated Master Nodes Count is at least 3.
Using AWS CLI:
Run the following command to describe the OpenSearch domain configuration:
aws opensearch describe-domain --domain-name <domain-name>
In the output, check the DedicatedMasterNodes field:
{
"DomainStatus": {
"DomainName": "your-domain-name",
"ClusterConfig": {
"DedicatedMasterNodes": 3
}
}
}Ensure that the number of dedicated master nodes is 3 or higher.
Implementation Plan:
Using AWS Console:
Log in to the AWS Management Console
navigate to the Amazon OpenSearch Service.
Select the domain you wish to modify or create a new domain.
In the Cluster Configuration section
Enable Dedicated Master Nodes.
Set the Dedicated Master Nodes Count to 3 or higher.
Save the configuration changes.
Using AWS CLI:
If you are creating a new domain, use the following command to configure dedicated master nodes during domain creation:
aws opensearch create-domain --domain-name <domain-name> --cluster-config "DedicatedMasterEnabled=true, DedicatedMasterCount=3"
If the domain already exists, update the domain configuration to add dedicated master nodes:
aws opensearch update-domain-config --domain-name <domain-name> --cluster-config "DedicatedMasterEnabled=true, DedicatedMasterCount=3"
Backout Plan:
If increasing the number of master nodes causes issues (e.g., performance degradation or unexpected costs):
Identify the affected domain and review the current configuration.
Revert the number of master nodes to two by running:
aws opensearch update-domain-config --domain-name <domain-name> --elasticsearch-cluster-config DedicatedMasterCount=2
Verify that the OpenSearch domain is still functioning correctly and the changes have been reverted.