Cloudwatch at some point starts posting with a frozen timestamp. https://github.com/aws/amazon-cloudwatch-agent/issues/198 .
If we restart the agent, the problem goes away, at least for a bit of time.
I am wondering if anybody else is seeing this issue and if there is any known work around.
On a whim, I tried sending logs larger thank 250k but those logs are correctly suppressed and the next logs work fine. I've attempted to look for patterns for when the anomaly begins, but so far no luck.