EBS Snapshot copy throttle

The issue you're experiencing with EBS snapshot copies slowing down and only one snapshot being copied at a time while others remain in a pending state could be due to several factors:

1. Full copies vs. incremental copies: When you're re-encrypting snapshots with a new KMS key, this initially results in a full copy rather than an incremental copy. Full copies take longer and consume more resources than incremental copies. However, after the first full cross-region or cross-account copy with the target KMS key has completed, copies of subsequent snapshots of the same source volume will be incremental. The complete rules on incrementality are explained here: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-copy-snapshot.html

2. Concurrent copy limits: While you've set up your Step Function to run 20 snapshot copies at a time, AWS may have internal limits on the number of concurrent snapshot copy operations that can be performed, especially for full copies.

3. Resource constraints: The slowdown could be due to resource constraints in the AWS Region where you're performing the copies. This can happen if there's high demand for snapshot copy operations in that Region.

4. KMS throttling: Since you're re-encrypting the snapshots, there might be KMS API request limits affecting the process. KMS has service quotas that could potentially slow down the encryption process if exceeded. However, if your KMS keys are dedicated for use with EBS volumes and snapshots and not shared across other services, such as S3 or SQS, KMS throttling is highly unlikely to be a factor.

5. Snapshot size and data changes: Larger snapshots or those with significant changes since the last copy will take longer to copy and could contribute to the slowdown.

To address this issue, you could try the following:

1. Spread the copy operations across multiple Regions if possible.
2. Contact AWS Support to check if there are any account-specific limits on concurrent snapshot copy operations that could be adjusted.
3. Monitor your KMS API request quotas and consider requesting an increase if necessary.
4. Optimize your snapshot management strategy to reduce the size and frequency of full copies where possible.

Remember that snapshot copy times can vary based on several factors, and occasional slowdowns may occur due to the nature of the service and resource availability.
Sources
Copy an Amazon EBS snapshot - Amazon EBS
How Amazon EBS snapshots work - Amazon EBS
Amazon EBS Fast Snapshot Restore (FSR) is now available for shared EBS Snapshots