Retention not working correctly

Question

I have set Kafka retention for 1 hour by the configurations per suggestions on every public doc. But I can't make it work correctly. I can make sense that it is trying to do as per my configurations, like when I set for 1 hour 60-70 percent data seems to get deleted and if I set it to 12 hours, again 60-70 percent get deleted. My requirement is, I want to delete all data and retention should delete all data as per the logic. Any ideas how can I make sure that there are no loose ends? All data should follow retention and not just some percent of it.

Answer

If you're experiencing issues with Kafka retention not deleting all data as expected, there are a few things you can check to ensure there are no loose ends:

- You may verify that the server/cluster level retention policy parameters are same as topic level configuration because if a topic is created with a different retention setting than the one you intend to use, Kafka will retain data based on the topic's retention policy instead of the server/cluster configuration.

- Also make sure that the 'log.cleanup.policy' configuration is set to 'delete' (the default) rather than 'compact'. The 'compact' policy retains the latest value for each key and may interfere with data deletion.

If the above suggestions doesn't help you, in that case you may reach to Support team via support case with more details such as current configuration, cluster ARN, which topic's documents are not getting deleted etc. and they will be happy to guide you further.

Retention not working correctly

Relevant content