How do I configure the retention period for my MSK cluster?

Short description

Use the Amazon MSK data retention period to control how much data you store in your MSK cluster. When you set a retention period, you can automatically delete old messages that you don't need, and you might reduce your storage costs and improve cluster performance.

By default, the Amazon MSK data retention period is 7 days. However, you can modify the retention period to meet your requirements. If you don't correctly configure the retention period, then Amazon MSK might not delete your data in the correct timeframe.

Apache Kafka divides topic partitions into segments. The default maximum segment size is 1 GB. To change the maximum size, update log.segment.bytes at the broker level or segment.bytes at the topic level. For more information, see Updating broker configs and Updating default topic configuration on the Apache Kafka website.

Apache Kafka also uses log.roll.ms to control a segment's lifecycle. These settings determine a segment's maximum age, and the default value is 168 hours. Apache Kafka only creates new segments when segments.bytes or log.roll.ms reach the configured limit value. Once Apache Kafka creates the new segment, the older segments are then inactive. Then, MSK deletes older segments when the new segments reach the configured retention.ms value.

Note: Only inactive segments can be deleted.

Resolution

To configure your retention period, modify your retention.ms parameter and your segment.ms setting.

For example, if you set the value of segment.ms to 24hrs and retention.ms to 24hrs, then Amazon MSK deletes the data after 48 hours. The segment is then inactive, depending on whether the segment first meets the 24 hours timeframe or the 1 GB data limit.

To delete the data after 24 hours, reduce the default segment.ms value to 1hr and retention.ms to 23hrs. Amazon MSK deletes the data after 24 hours or when the segment reaches the configured size value, whichever comes first.

To calculate the current size of all topics, run the following command on the Apache Kafka command line interface (CLI) in Amazon MSK:

./kafka-topics.sh --command-config client.properties --bootstrap-server $BK --list |
xargs -I{} sh -c "echo -n '{} - ' && 
./kafka-log-dirs.sh --command-config client.properties --bootstrap-server $BK --topic-list {} --describe |
grep '^{'  |
jq '[ ..|.size? | numbers ] | add' |
numfmt --to iec " |
tee /tmp/topics-by-size.list

Note: The preceding command uses the Apache Kafka CLI to list all the topics, then calculates and returns the size of each topic.

Example output:

BrandNewTopic - 0
CrossAccountMskTopic - 0
NewTopicwithNewInstance - 0
SameAccountMskTopic - 0
Test - 0
__amazon_msk_canary - 4.6M
__consumer_offsets - 33M
dataloss - 1.8K
datalossB1 - 999
provisioned - 0

The following are best practices for storage utilization:

• Check your log.segments.bytes or log.roll.ms value and reduce the value for your segment. If you set a smaller value, then you can create segments frequently and deactivate older segments.
• Fine tune the segment.ms and segment.bytes parameters to balance storage efficiency and broker performance. Don't set the paraments too low because that can create additional CPU overhead and might strain your broker resources.

Note: log.segments.bytes and log.roll.ms are broker level segment settings. segment.bytes and segment.ms are topic level segment settings. Topic level settings takes high precedence than broker level settings.

Related information

retention.ms on the Apache Kafka website