Profile Applicability:
Level 1
Description:
Amazon Kinesis Data Streams are used for real-time data streaming and processing. Data in Kinesis streams is retained for a configurable retention period, which determines how long the data is stored in the stream before it is automatically deleted. The default retention period for Kinesis streams is 24 hours, but it can be increased up to 7 days.
Setting an adequate data retention period ensures that your data is available for the time required to perform processing, troubleshooting, and analysis. It also ensures that the stream can handle data replay and reprocessing in case of failures or delays.
Rationale:
Data Availability: The retention period determines how long data is available in the stream. A longer retention period ensures that your applications can process data without worrying about losing information prematurely.
Business Continuity: If data processing or analysis fails, the ability to replay data from Kinesis streams ensures business continuity.
Compliance: Some regulatory requirements may require retaining data for specific periods. Configuring the correct retention period helps meet compliance obligations.
Efficient Resource Management: Setting an adequate retention period prevents unnecessary data storage, ensuring efficient resource management while meeting business needs.
Impact:
Pros:
Improved Data Accessibility: Ensures that data remains accessible in the stream for the required amount of time.
Compliance: Meets regulatory requirements for data retention in certain industries.
Reduced Data Loss: Enables data reprocessing in case of missed or failed processing attempts.
Business Continuity: Provides data for troubleshooting and error recovery.
Cons:
Increased Costs: A longer retention period will incur higher costs for storing the data in the stream.
Potential Overhead: Retaining data for a long time can lead to increased storage costs, especially if the stream receives a high volume of data.
Default Value:
By default, Amazon Kinesis Data Streams retain data for 24 hours. You can modify the retention period to extend the duration, up to 7 days.
Pre-requisite:
AWS IAM Permissions:
kinesis:DescribeStream
kinesis:IncreaseStreamRetentionPeriod
kinesis:DecreaseStreamRetentionPeriod
AWS CLI installed and configured.
Kinesis Stream should be active and accessible.
Retention Period Policy should be defined based on business and regulatory requirements.
Remediation:
Test Plan:
Using AWS Console:
Sign in to the AWS Management Console.
Navigate to Amazon Kinesis under Services.
In the Kinesis Dashboard, select Data Streams.
Choose the stream for which you want to check or modify the retention period.
Under Stream Details, look for the Retention Period setting.
If the Retention Period is less than the required duration, modify it.
To change the Retention Period, click Modify, set the desired retention period (up to 7 days), and save the changes.
Using AWS CLI:
To check the current retention period for a Redshift stream, run:
aws kinesis describe-stream --stream-name <stream-name> --query 'StreamDescription.RetentionPeriodHours'
If the retention period is insufficient, to increase the retention period (for example, to 48 hours), run:
aws kinesis increase-stream-retention-period --stream-name <stream-name> --retention-period-hours 48
To confirm the new retention period, run:
aws kinesis describe-stream --stream-name <stream-name> --query 'StreamDescription.RetentionPeriodHours'
Implementation Steps:
Using AWS Console:
Log in to the AWS Management Console and navigate to Amazon Kinesis.
In the Kinesis Dashboard, select Data Streams.
Choose the stream you want to modify.
Under Stream Details, locate the Retention Period.
If the current retention period is not adequate, click Modify and increase it to the desired retention period (up to 7 days).
Save the changes and confirm the new retention period.
Using AWS CLI:
To modify the retention period of a Kinesis stream, run the following command to extend it:
aws kinesis increase-stream-retention-period --stream-name <stream-name> --retention-period-hours <desired-retention-period-in-hours>
To confirm the updated retention period, use:
aws kinesis describe-stream --stream-name <stream-name> --query 'StreamDescription.RetentionPeriodHours'
Backout Plan:
Console Process
Go to the Amazon Kinesis Console:
Open the Amazon Kinesis console and select your stream.
Configure Retention Period:
Navigate to the Stream details section.
Adjust the Data Retention Period to your desired time (minimum: 24 hours, maximum: 7 days).
CLI Process
Update Retention Period:
aws kinesis increase-stream-retention-period \ --stream-name <stream-name> \ --retention-period-hours <hours>
Verify Retention Settings:
aws kinesis describe-stream --stream-name <stream-name>
Check the RetentionPeriodHours field in the output.
Note:
Cost Management: Keep track of the storage costs associated with increased retention periods and adjust accordingly based on your organization’s data storage policy.
Testing: It is advisable to test retention period changes in a non-production environment to ensure that no performance or operational issues arise when increasing retention periods.