Skip to content

Able to store/write object to FSx even if the cmk kms key attached is disabled/deleted

0

Hi,

We created a customer-managed KMS key (CMK) and associated it with an Amazon FSx for Windows File Server instance. After the FSx file system was up and running, we disabled the CMK and revoked all its grants.

Surprisingly, even after 24 hours FSx is still allowing us to write/store files to the file system, even though the key is disabled and no valid grants exist.

We expected that any write operations that require encryption would fail once the key became unavailable.

Our questions:

  1. Why is FSx still functioning as if the CMK is available?

  2. Does FSx cache data keys or encryption context that allow it to keep operating temporarily?

  3. When will FSx stop accepting writes if the CMK remains disabled or is scheduled for deletion?

Few observation: I was able to create backup of the FSx

Any clarity on how we can remove key from caching.

Thanks!

asked a year ago181 views

2 Answers
0

Amazon FSx for Windows File Server is based on EC2 instances running the Windows Server operating system and with data stored on EBS volumes (virtual hard drives). The rules for how KMS keys are used are essentially the same as for EBS volumes attached to self-managed EC2 instances.

Access to the KMS key is only needed for existing EBS volumes when attaching them to an EC2 instance or when powering on a powered-off (or hibernating) EC2 instance. The EC2 hypervisor that runs the instance keeps the data encryption keys (DEKs, decrypted with the respective KMS key) in memory for as long as the instance is running and the EBS volume, so as to be able to read from and write to the EBS volume.

Disabling or deleting the KMS key will only stop FSx from powering on or replacing the EC2 instances that run the file system, because the instances wouldn't be able to decrypt the DEKs with the KMS key that is no longer available.

EXPERT

answered a year ago

0

Based on my experience working with KMS-integrated services like Amazon FSx, what you’re observing is expected behavior, though a bit surprising at first glance.

Even after you disable the CMK and revoke grants, FSx may continue functioning for a period because it caches the data encryption keys (DEKs) that were originally generated using your CMK. These cached DEKs allow FSx to continue encrypting and decrypting data without making fresh KMS calls, which explains why write operations and even backups still work temporarily.

This caching mechanism is intentional; it helps avoid disruptions in scenarios where KMS might be briefly unavailable. However, suppose the CMK remains disabled or scheduled for deletion. In that case, FSx will eventually fail when it needs to generate new DEKs, rotate keys, or access metadata that requires re-authentication with the CMK.

In my past experience, the exact timing of failure isn't guaranteed, but issues typically begin to surface after a few hours to days, depending on workload patterns and key reuse. For example, write operations might still succeed using cached DEKs, but certain administrative actions or file system operations (like restoring from backup) may begin to fail sooner.

Regarding clearing the cache, there is no customer-facing mechanism to flush or reset the key cache within FSx. The only guaranteed way to trigger key re-validation is to restart the file system or wait until FSx attempts a key operation that requires an active CMK.

Suppose your goal is to test the failure behavior or enforce encryption restrictions thoroughly. In that case, I suggest creating a new FSx instance with a test CMK, disabling the key, and carefully observing the timing and failure points. But in production, I’d recommend never disabling or revoking access to a CMK actively used by a managed service unless it’s being rotated in a planned and reversible way.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.