Best practice to use AWS Secrets Manager with EKS and rotate regularly

Question

We are currently considering externalizing secrets for Kubernetes (EKS) by utilizing AWS Secrets Manager. Our proposed approach involves using the K8s Secrets Store CSI Driver and ASCP to retrieve secrets from Secrets Manager and then pass them as environment variables to the container.

Additionally, we have identified other potential options:

- Utilizing the Secrets CSI driver and mounting secrets - However, we are unsure about the correct method for utilizing the mounted secrets and how to integrate them effectively.

- Making direct API calls from the Java application.

Are there any alternative approaches we should consider?

Furthermore, we have a requirement to automatically rotate these secrets at regular intervals, such as every 7 days. It is crucial that the rotation remains synchronized with the secrets used by the application. We are seeking an optimal approach for achieving this synchronization with zero downtime. The secret auto rotation feature provided by the CSI driver relies on polling, but it's important to note that AWS charges for each API call. Hence, we aim to minimize API calls to reduce costs while ensuring zero downtime. Setting the polling interval too low would increase costs, while setting it too high may result in downtime if rotation fails to occur within the expected time frame. Are there alternative methods to address this concern? If not, how can we appropriately determine the optimal polling interval?

AWS-Harrison · Answer

Consider switching from a pull model, to a push model. Create an automated workflow such that:
1. Create an entirely new secret with the new data, leaving the old secret intact.
2. Update the PodSpec of the Deployment to point to the new secret.
3. Recycle your pods at a controlled rate. I recommend using a PodDisruptionBudget to prevent downtime here.
4. Validate all pods have been updated and are using the new secret successfully. If this fails, switch back to the old secret as an error handled.
5. Delete the old secret after such time that you feel comfortable.

If we automate this process with something like AWS Step Functions, then you never need to set up a polling API call.

Best practice to use AWS Secrets Manager with EKS and rotate regularly

Relevant content