We have a CloudHSM cluster with 2 HSM in different zones (azA and azB). We noticed that once in several days (sometimes once in 4 days, sometimes daily), the HSMs are being destroyed and then created back one by one, by the cloudhsm.amazonaws.com service. We thought that this undocumented behavior is expected during backup creation but according to audit logs, there are HSMs that are performing several (daily) backups before are being destroyed/rotated.
In the next graph, you can notice that HSM count raises to 3 during this "rotation" event.
Could anyone explain if this is an expected behavior and what is the reason for this "rotation"? HSMs are not under load or temperature pressure at all.