Profile Applicability:

  • Level 2

Description:

An FSx bucket is typically associated with Amazon FSx for Lustre, a fully managed file system optimized for high-performance computing. In the context of FSx, a bucket is often used to store and manage data from Amazon S3 or serve as a target for file system exports. Ensuring the creation and configuration of an FSx bucket ensures proper storage management and facilitates data access between Amazon FSx and S3.

Rationale:

Creating an FSx bucket ensures:

  • Proper storage for files processed by FSx for Lustre

  • Seamless data exchange between Amazon FSx and Amazon S3

  • Performance optimization for high throughput workloads

  • Secure and efficient storage management for backup, recovery, and processing

Default Value:

An FSx bucket is not created by default when you create an FSx file system. It must be manually set up and configured, typically with an S3 integration for data management.

Impact:

Pros:
 • Ensures efficient storage management for FSx file systems
• Facilitates seamless integration between FSx and Amazon S3
• Optimizes storage for high-throughput and data-intensive workloads
 • Enables automatic export of file system data to an S3 bucket for backup and processing

Cons:
 • Requires manual configuration and permissions management
• Potential for misconfiguration if IAM roles or policies are not correctly set up
 • Additional AWS storage costs for storing data in the FSx bucket

Pre-requisites:

IAM Permissions Required:
 s3:CreateBucket, fsx:CreateFileSystem, fsx:AssociateFileSystem
 ec2:DescribeInstances, ec2:DescribeVolumes
 Full access to create and configure FSx and S3 buckets

Remediation:

Test Plan:

Using AWS Console:
 • Log in to the AWS Management Console
 • Navigate to FSx > File systems
 • Ensure you have an existing FSx for Lustre file system
 • Go to the Amazon S3 section and link the file system to an S3 bucket, or create a new bucket if necessary
 • Review the IAM roles and permissions to ensure the FSx file system has the correct access to the bucket
 • Confirm that the FSx file system is correctly writing and reading data from the linked S3 bucket

Using AWS CLI:

aws fsx describe-file-systems \
  --file-system-id fs-xxxxxxxxxxxxxxxxx
aws s3api create-bucket \
  --bucket <bucket-name> \
  --region <region>

Implementation Plan:

Using AWS Console:
 • Navigate to FSx > File systems and select your FSx for Lustre file system
 • Under Storage or S3 Integration, choose to Link to S3
 • Create a new bucket or select an existing S3 bucket for storage integration
 • Review IAM roles and permissions for FSx to access the S3 bucket
 • Ensure export tasks are configured to move data between FSx and the S3 bucket (optional)
 • Confirm the setup by performing a test to read and write data between FSx and S3

Using AWS CLI:
 Step 1: Create the S3 bucket for FSx integration

aws s3api create-bucket \
  --bucket <bucket-name> \
  --region <region>

Step 2: Link the FSx file system to the S3 bucket

aws fsx associate-file-system-with-s3 \
  --file-system-id fs-xxxxxxxxxxxxxxxxx \
  --s3-bucket-name <bucket-name> \
  --s3-key-prefix <prefix>

Step 3: Verify the file system and S3 bucket linkage

aws fsx describe-file-systems \
  --file-system-id fs-xxxxxxxxxxxxxxxxx

Step 4: Test read/write access to the FSx bucket

aws s3 cp file.txt s3://<bucket-name>/testfile.txt

Backout Plan:

Using AWS Console:
 • Unlink the FSx file system from the S3 bucket under the FSx for Lustre > File systems section
• Delete the FSx file system or reconfigure it to point to another storage location
 • Remove the S3 bucket if no longer needed

Using AWS CLI:
 To unlink the FSx file system from the S3 bucket:

aws fsx disassociate-file-system-from-s3 \
  --file-system-id fs-xxxxxxxxxxxxxxxxx

To delete the S3 bucket (if no longer needed):

aws s3api delete-bucket \
  --bucket <bucket-name>

References: