Description: 

Elasticsearch Service(ES) is a managed service that makes it easy to deploy, operate, and scale Elasticsearch, a popular open-source search, and analytics engine. Amazon ES also offers security options, high availability, data durability, and direct access to the Elasticsearch API. It is a popular open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis.


Rationale:

By enabling logs in the Elastic Service log, we examine the slow queries and updates. With this knowledge, you can optimize the queries or tailor your deployment for indexing. This can also be integrated with Cloud Watch Logs and ES to send slow logs to a different Amazon ES domain and monitor your domain’s performance.


Impact: 

Enabling logs will help in

  • Determining whether there is proper communication between the Elasticsearch and the application.

  • We can examine the queries that are hitting Elasticsearch and validate them.

  • Logging can provide crucial information about index/cluster health, and thus help maintain the cluster.

  • Can improve the performance issues caused by specific queries or due to changes in cluster usage


Default Value: 

By default, logging is not enabled when launching the Elasticsearch Service.


Pre-Requisite:

The Elasticsearch should be configured earlier and deployed with VPC Subnets and Security Groups etc, to set up the slow log Services.


Remediation:


Test Plan:

  1. Log in to the AWS Management Console.

  2. Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es

  3. Click on the  ES domain which we want to enable the logs. 

  4. Go to the logs section inside our domain

  5. Check the status of the Search Slow Logs and Index Slow Logs and Error Logs

  6. If the status is disabled, it is confirmed that logging has not been enabled in the Elasticsearch Service


Implementation Steps:

  1. Log in to the AWS Management Console.

  2. Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es

  3. Click on the  ES domain which we want to enable logs.

  4. Click on the Logs tab

  5. To enable Search Slow logs

    1. Click on setup

    2. Under the CloudWatch Logs log group Choose whether you want to create a new log group

      or Use the existing log group

    3. Under the Specify Resource Access Policy choose whether to create a new policy or select an existing policy

    4. Save the settings by clicking Enable

  6. To enable Index slow logs

    1. Click on setup

    2. Under the CloudWatch Logs log group Choose whether you want to create a new log group

      or Use the existing log group 

    3. Under the Specify Resource Access Policy choose whether to create a new policy or select an existing policy

    4. Save the settings by clicking Enable

  7. To enable Error logs

    1. Click on setup

    2. Under the CloudWatch Logs log group, Choose whether you want to create a new log group

      or Use the existing log group 

    3. Under the Specify Resource Access Policy choose whether to create a new policy or select an existing policy

    4. Save the settings by clicking Enable 

  8. Repeat steps 1 to 8 till logs have been enabled in Elasticsearch Service domains in every region


Backout Plan: 

  1. Log in to the AWS Management Console.

  2. Navigate to Elasticsearch (ES) dashboard at https://console.aws.amazon.com/es

  3. Click on the  ES domain which we want to enable the slow log.

  4. Click on Logs

  5. To disable Search Slow logs click on Disable


  6. To Disable Index slow logs, click on Disable

  7. To Disable Error logs, click on Disable



Note:

  • If you plan to enable multiple logs, we recommend publishing each to its log group. This separation makes the logs easier to scan.

  • Choose an access policy that contains the appropriate permissions, or create a policy using the JSON that the console provides.

  • Enabling Log Publishing (Console), the Amazon ES console, is the simplest way to enable the publishing of logs to CloudWatch. To enable log publishing to CloudWatch (console)

    • Go to https://aws.amazon.com, and then choose Sign In to the Console.

    • Under Analytics, choose Elasticsearch Service.

    • In the navigation pane, under My domains, choose the domain that you want to update.

    • On the Logs tab, choose Enable for the log that you want.

    • Create a CloudWatch log group, or choose an existing one.


Reference: