Has anyone been able to change from AWS KMS to a CMK in Secrets manager in bulk

You're correct that AWS Secrets Manager currently doesn't provide a native bulk operation to change encryption keys from the AWS managed key (aws/secretsmanager) to a customer managed key (CMK).

The AWS Well-Architected recommendations do suggest using a customer managed key in certain scenarios, particularly when you need to:

• Access secrets from another AWS account
• Apply specific key policies to the encryption key
• Rotate your own KMS key

However, it's worth noting that for most standard use cases, the AWS managed key aws/secretsmanager is actually recommended and has no additional cost, unlike customer managed keys.

To change encryption keys in bulk, you would need to create a custom automation using AWS CLI or SDK scripts that:

1. List all your secrets
2. For each secret, call the UpdateSecret API with the new KMS key ID parameter

When you change the encryption key, Secrets Manager re-encrypts the AWSCURRENT, AWSPENDING, and AWSPREVIOUS versions with the new key. Note that if you don't have permission to decrypt with the previous key, existing versions won't be re-encrypted.

If you have custom-labeled versions of secrets, you'll need to recreate those versions using the new key, as they won't be automatically re-encrypted during the key change process.
Sources
Change the encryption key for an AWS Secrets Manager secret - AWS Secrets Manager
UpdateSecret - AWS Secrets Manager
Encryption best practices for AWS Secrets Manager - AWS Prescriptive Guidance

You can loop through all the existing secrets with the AWS CLI that you can run in AWS CloudShell. Make sure to start CloudShell in the same region where your Secrets Manager secrets reside.

First, discover the "KmsKeyId" value that is returned for a secret that is currently using the old key. You'll use the discovered in this stage to apply the change only to those secrets that are using the old AWS managed key. For example:

aws secretsmanager describe-secret --secret-id SOME_OLD_SECRET_NAME

The response will show the current KMS key identifier in the KmsKeyId property. It may also show as an ID or alias, such as alias/aws/secretsmanager. Capture the value exactly as shown in the "KmsKeyId" field of the response:

{
    "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:SOME_OLD_SECRET_NAME-Ca8JGt",
    "Name": "SOME_OLD_SECRET_NAME",
    "Description": "My old secret",
    "KmsKeyId": "arn:aws:kms:us-east-1:000000000000:key/OLDKEY1-90ab-cdef-fedc-ba987EXAMPLE",
    "RotationEnabled": false,
   ...

Then prepare a loop command in a text editor. Replace the old key ARN placeholder arn:aws:kms:us-east-1:000000000000:key/OLDKEY1-90ab-cdef-fedc-ba987EXAMPLE with the "KmsKeyId" value captured above. Replace the new key ARN placeholder arn:aws:kms:us-east-1:000000000000:key/NEWKEY1-90ab-cdef-fedc-ba987EXAMPLE with the new CMK you want to use. When both placeholders are replaced, copy the commands in the CloudShell prompt to change the key for every secret currently using the old key.

secret_names=$(aws secretsmanager list-secrets | jq -r '.SecretList[] | select(.KmsKeyId == "arn:aws:kms:us-east-1:000000000000:key/OLDKEY1-90ab-cdef-fedc-ba987EXAMPLE") | .Name')
echo "${secret_names}" | while IFS= read -r secret; do
  aws secretsmanager update-secret --secret-id "${secret}" --kms-key-id arn:aws:kms:us-east-1:000000000000:key/NEWKEY1-90ab-cdef-fedc-ba987EXAMPLE
done

Note that as mentioned in documentation, https://docs.aws.amazon.com/secretsmanager/latest/userguide/manage_update-encryption-key.html, this procedure will change the KMS key for the standard version labels AWSCURRENT, AWSPENDING, and AWSPREVIOUS. Typically, these are all the version you'd have, but if you're using any custom labels, you'll need to add it to the end of the update-secret command. For example: --version-stage MyCustomLabel