Is log cleaner logging anything on AWS MSK?
I use AWS MSK cluster with brokers logging turned on to CloudWatch. I can see brokers logs. We have some topics with cleanup.policy=compact and some with cleanup.policy=delete. The system is running on the new cluster for about 2 weeks now.
From my research (e.g. https://zendesk.engineering/an-investigation-into-kafka-log-compaction-5e520f4291f0) I see that kafka should run log cleaner (obviously) and there should be some traces in logs of this activity. However in my CloudWatch log group I cannot find a word "cleaner" or "cleaned" and I cannot find any trace of log cleaner running.
Is it something wrong with my logging settings? Or is log cleaner not running at all (this is suspicious, because we have a lot of messages eligible for cleanup but still not cleaned)?
Hello there, in order to verify from the logs whether files related to partitions are getting deleted or not you can look for the something similar as below
Deleted log LOG_DIR/<topic>/00000000022134078627.log.deleted Deleted time index LOG_DIR/<topic>/00000000022134078627.timeindex.deleted
When retention kicks in you will be noticing deletion of log and time index files in the broker logs. That indicates retention is successfully triggered and deleted old log segments.
Thank you! Yes I have noticed such logs, but still, the disk usage is constantly growing, while we have about 99% of records eligible for deletion. This records are in __consumer_offsets topic where we commit last message every 100 millis even if there are no new messages for hours.
Scaling a clusterasked 3 years ago
Is log cleaner logging anything on AWS MSK?asked 2 months ago
Amazon MSK Authentication and Authorizationasked 6 months ago
Clarifying MSK data transfer pricing within region or AZAccepted Answerasked 3 years ago
Storage Capacity on MSK maxed outasked a month ago
Unable to use IAM permissions to access MSK Brokersasked a year ago
Logging with CloudWatch vs. ElasticSearch/KibanaAccepted Answerasked 5 years ago
I want consolidated application logs running on AWS ECS with microservices architectureasked a month ago
Automatically update brokerstring in MSKAccepted AnswerEXPERTasked 2 years ago
Brokers Connection string doesn't show creating cIuster with IAM AccessAccepted Answerasked a year ago