Rate Exceeded Throttle error

As we are using cloudwatch appender to write the logs to cloudwatch from EKS Pods Some time we get com.amazonaws.services.logs.model.AWSLogsException: Rate exceeded (Service: AWSLogs; Status Code: 400; Error Code: ThrottlingException; Request ID: 24102722-da9f-4b0a-a324-903f1e45a01b; Proxy: null)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)

at at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)

at at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)

at at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)

at at com.amazonaws.services.logs.AWSLogsClient.doInvoke(AWSLogsClient.java:3448)

at at com.amazonaws.services.logs.AWSLogsClient.invoke(AWSLogsClient.java:3415)

at at com.amazonaws.services.logs.AWSLogsClient.invoke(AWSLogsClient.java:3404)

at at com.amazonaws.services.logs.AWSLogsClient.executeDescribeLogStreams(AWSLogsClient.java:1548)

at at com.amazonaws.services.logs.AWSLogsClient.describeLogStreams(AWSLogsClient.java:1517)

at at io.github.dibog.AwsCWEventDump.findLogStream(AwsCWEventDump.java:144)

at at io.github.dibog.AwsCWEventDump.openStream(AwsCWEventDump.java:108)

at at io.github.dibog.AwsCWEventDump.log(AwsCWEventDump.java:170)

at at io.github.dibog.AwsCWEventDump.run(AwsCWEventDump.java:223)

at at java.base/java.lang.Thread.run(Thread.java:829)

06:52:12,745 |-INFO in io.github.dibog.AwsLogAppender[cloud-watch] - creating log stream '/usermanagementservices'

07:53:51,948 |-INFO in c.q.l.co.rolling.helper.RenameUtil - Renaming file [logs/app.log] to [logs/app.16-06-2023.3.log17845009121291222.tmp]

07:53:51,948 |-INFO in ch.qos.logback.core.rolling.helper.Compressor - GZ compressing [logs/app.16-06-2023.3.log17845009121291222.tmp] as [logs/app.16-06-2023.3.log.gz]

07:53:51,953 |-INFO in c.q.l.core.rolling.helper.TimeBasedArchiveRemover - Removed 0 Bytes of files