iCompaas Support

Description:

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It is the most popular search engine and is commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases. With Elastic search service domains, you get an endpoint that you resolve through DNS, which enables communication with the Elasticsearch cluster.

Rationale:

Public domains are accessible from any internet-connected device, VPC domains require some form of VPN or proxy. Because of their logical isolation, domains that reside within a VPC have an extra layer of security compared to domains that use public endpoints.

Impact:

Compared to public domains, VPC domains display less information in the console. You can't apply IP-based access policies to domains that reside within a VPC because security groups already enforce IP-based access policies.

Default Value:

AWS recommends VPC Access over the public access while creating the Elasticsearch Domain.

Audit:

• via Management Console

1. Sign in to the AWS Management console

2. Navigate to Amazon OpenSearch (successor to Elasticsearch) at https://console.aws.amazon.com/esv3/

3. Choose an ES domain required from the list of available domains

4. Look for the Endpoint attribute of the desired domain.
   

5. If the endpoint value is set to Internet, it represents a publicly accessible domain.
6. If the endpoint value is set to VPC, it represents a domain residing in an AWS VPC.

• via CLI

1. Get the list of ES domains in a region using list-domain-names command.

   aws es list-domain-names
     --region <value>

2. Use describe-elasticsearch-domain get the Access endpoint of the domain

   aws es describe-elasticsearch-domain
     --domain-name <value>
     --region <value>
     --query 'DomainStatus.Endpoint'

3. If the domain is publicly accessible, it returns a public endpoint URL.

4. If the domain is in VPC, it returns a NULL value

Remediation:

Pre-Requisite:

•  To migrate AWS Elasticsearch domains from public access to VPC access, you must unload the existing data from the ES domain to Amazon S3 then upload this data in a new ES cluster, launched within a Virtual Private Cloud. 

OR

• Take the snapshot of the source domain using

  curl -XPUT 'domain-endpoint/_snapshot/repository-name/snapshot-name'

 Implementation steps:

• via Management Console

1. Sign in to the AWS Management console

2. Navigate to Amazon OpenSearch (successor to Elasticsearch) at https://console.aws.amazon.com/esv3/

3. Choose the domain that has public access 

4. Copy all the selected cluster configuration information such as Instance count, Instance type, Dedicated master instance type, Dedicated master instance count, Storage Type, EBS volume size, Access policy, etc.

5. On the AWS ES dashboard, click the Create Domain button to launch a new Elasticsearch domain.
   

6. Give a unique name for the new ES Domain
   

7. Set the new domain parameters using configuration details copied earlier.
8. Under Network section, choose VPC Access to launch the domain within a VPC. And then choose the desired VPC, subnet, and the security groups from the respective drop-downs.
   
9. Under Access policy section, choose Configure domain level access policy to import policy from an existing ES Domain or use Visual editor/JSON to edit it to meet the needs of the new ES Domain.
   
10. Click Confirm and create to launch the new AWS Elasticsearch domain within the specified AWS VPC.
11. After successful creation of the new ES Domain, upload the data from the source domain to the new ES domain by restoring the manual snapshots from the old instance (backup created as a pre-requisite). Do this by SSH-ing into the ES Server.

• via CLI

1. Create a new domain using create-elasticsearch-domain to relaunch the selected ES Domain into an AWS VPC

   aws es create-elasticsearch-domain
     --region <value>
     --domain-name <value>
     --elasticsearch-version <value>
     --elasticsearch-cluster-config InstanceType=<value>, InstanceCount=<value>
     --ebs-options EBSEnabled=<value>, VolumeType=<value>, VolumeSize=<value>
     --vpc-options SubnetIds=<value>, SecurityGroupIds=<value>

   
   

2. After successful creation of the new ES Domain, upload the data from the source domain to the new ES domain.

3. To restore a snapshot, perform the following steps:

   Identify the snapshot you want to restore. To see all snapshot repositories, run the following command:

   curl -XGET 'domain-endpoint/_snapshot?pretty'

   After you identify the repository, run the following command to see all snapshots:

   curl -XGET 'domain-endpoint/_snapshot/repository-name/_all?pretty'

   Note: Most automated snapshots are stored in the cs-automated repository. If your domain encrypts data at rest, they're stored in the cs-automated-enc repository. If you don't see the manual snapshot repository you're looking for, make sure you registered it to the domain.
   
   
   To restore a snapshot, run the following command:

   curl -XPOST 'domain-endpoint/_snapshot/repository-name/snapshot-name/_restore'

   Due to special permissions on the OpenSearch Dashboards and fine-grained access control indices, attempts to restore all indices might fail, especially if you try to restore from an automated snapshot. The following example restores just one index, my-index, from 2020-snapshot in the cs-automated snapshot repository:

   curl -XPOST 'domain-endpoint/_snapshot/cs-automated/2020-snapshot/_restore' -d '{"indices": "my-index"}' -H 'Content-Type: application/json'

   
   

   Backout Plan:

   • via Management Console

   If you launch a new domain within a VPC, you can't later switch it to use a public endpoint and vice versa.

   Follow the steps to delete the domain

   1. Choose the domain created with the VPC Endpoint 

   2. Click the Delete button on the top-right to revoke changes to use publicly accessible domain.
      

   
   

   • via CLI

   To revoke changes and use the publicly accessible domain instead of the In VPC domain created, delete the VPC Access domain.

   aws es delete-elasticsearch-domain
     --region <value>
     --domain-name <value>

   
   

   
   

Note:

• After you place a domain within a VPC, you can't move it to a different VPC. However, you can change the subnets and security group settings.

• You can't launch your domain within a VPC that uses dedicated tenancy. You must use a VPC with tenancy set to Default.

• You can use manual snapshots to migrate individual indices or entire OpenSearch Service domains. You can migrate data to a domain in the same account or to a domain in a different account.

• If you're migrating to another AWS account, attach a policy to the source s3 bucket that grants cross-account permissions to the destination.

Reference:

Launching your Amazon OpenSearch Service domains within a VPC - Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) 

create-elasticsearch-domain — AWS CLI 1.20.58 Command Reference 

https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-migrate-domain/