Can AWS support re-key a MSK cluster to an AWS managed KMS key without disrupting the cluster?


I have a MSK cluster that has an encryption at rest key that has been deleted. The cluster is still operating, but I cannot make changes to the cluster (to add SASL/SCRAM auth) due to the missing key. I was wondering if there was a way AWS support could rotate the AWS managed Kafka key in (much like the annual key rotation) and "heal" the state of the cluster so I would not have to tear it down and re-create it.


1 Answer


On the Encryption at rest: It is not possible for an AWS Managed Key to go missing. You cannot change any properties of AWS managed keys, rotate them, change their key policies, or schedule them for deletion. So, I don't suspect this issue is related to AWS Managed Key (aws/kafka) that is used for encryption at rest.

I suspect the problem is in the way SASL/SCRAM authentication was setup in your Amazon MSK cluster, you can use AWS Secrets Manager to store and secure sign-in credentials. After setting up a secret, you can connect your client to the cluster using the credentials.

Also, there is a possibility you are trying to make changes to the cluster while it is processing a previous update. MSK clusters updates can take a few mins complete.

I would suggest logically compare your cluster setup to the steps mentioned in this workshop:

Further reading:


profile pictureAWS
answered a year ago
  • Hello Rama,

    I think it was a customer KMS key that was deleted, but it seems it has left the cluster in a working, but unchangeable state. Is it possible for AWS support to change the encryption at rest key to the standard AWS Kafka key while keeping the cluster running with the same broker and zookeeper urls? I am hoping to get the cluster into a state where it does not throw this error when making changes in the UI:

    Error updating security settings Amazon MSK wasn’t able to update the security settings. Wait a few minutes and then try again. API response To prevent further degradation of the cluster, Amazon MSK rejected the operation. The following required resource(s) are no longer accessible to the Amazon MSK service: [old key ARN here]. We recommend that you immediately migrate any critical data from this cluster to a new cluster, and that you stop all production usage of this cluster. For more details or assistance, contact AWS Support.

  • Hi,

    Deleting an AWS Customer KMS key is destructive and potentially dangerous. It deletes the key material and all metadata associated with the KMS key and is irreversible. After a KMS key is deleted, you can no longer decrypt the data that was encrypted under that KMS key, which means that data becomes unrecoverable.

    Because it is destructive and potentially dangerous to delete a KMS key, AWS KMS requires you to set a waiting period of 7 – 30 days. The default waiting period is 30 days. To recover the KMS key, you can cancel key deletion before the waiting period ends. After the waiting period ends you cannot cancel key deletion, and AWS KMS deletes the KMS key.

    So, in this case you could try and cancel the key deletion from the KMS console, if it is fully deleted, then do it as per the instructions- migrate to a new cluster and/or contact AWS support.


You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions