Automatically restart ECS service tasks/containers after secret rotation

Question

I'm using CDK to deploy a 3rd party application on ECS Fargate, that's configured to access a database via a secret in AWS Secrets Manager - using the direct integration from ECS something like the below:

```typescript
    this.secret = new secretsmanager.Secret(this, "Secret", {
      generateSecretString: {
        secretStringTemplate: JSON.stringify({
          database: "default",
          user: "me",
        }),
        generateStringKey: "password",
        excludePunctuation: true,
        passwordLength: 24,
      },
    });

...

const container = taskDefinition.addContainer("MyAppContainer", {
      containerName: "myapp",
      image: ecs.ContainerImage.fromEcrRepository(
        deployedImage.repository,
        deployedImage.imageTag,
      ),
      cpu,
      memoryLimitMiB,
      secrets: {
        SECRET_ENV_VAR: ecs.Secret.fromSecretsManager(this.secret, "password"),
      },
      ...
    });

this.fargateService = new ecs.FargateService(this, "FargateService", {
      taskDefinition: taskDefinition,
      ...,
    });
```

The question is, if I configure rotation on the Secrets Manager Secret, what's the best / most direct way to automatically restart the service's containers whenever the rotation happens? Since it's a 3rd party app, I don't have direct control over the code to implement client-side secret caching as described [here](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/rotate-database-credentials-without-restarting-containers.html): I think we have no choice but to pass it as an environment variable and restart the container when it changes.

Vivek V [AWS Hero] · Answer

I had implemented something similar using EventBridge to trigger a StepFunction for task restart , you can do something similar

Refer to my [blog post ](https://aws.plainenglish.io/automating-ecs-task-restarts-on-ecr-image-push-with-aws-step-functions-and-cdk-b7e7acf9b3b7)

```
const secretName = 'my-secret'; // Replace with your secret name
const rule = new events.Rule(this, 'SecretRotationRule', {
  ruleName: 'SecretRotationRule-' + secretName,
  eventPattern: {
    source: ['aws.secretsmanager'],
    detailType: ['AWS API Call via CloudTrail'],
    detail: {
      eventSource: ['secretsmanager.amazonaws.com'],
      eventName: ['RotateSecret'],
      requestParameters: {
        secretId: [secretName],
      },
    },
  },
});

rule.addTarget(new targets.SfnStateMachine(stateMachineL2));

```

Kauê Fernandes Braz · Answer

Have you considered using Amazon EventBridge to automate the restart of your ECS service when the secret is rotated?

With EventBridge, you can listen for "RotateSecret" events from Secrets Manager and trigger an ECS service update, ensuring your containers always load the latest credentials.

Automatically restart ECS service tasks/containers after secret rotation

Add your answer

Relevant content