ECS Service Connect Health Check

Question

We are using ECS Service Connect via a CloudFormation template using the ServiceConnectConfiguration property in the ECS Service configuration. This all works fine. However, we have noticed that when we scale out, the new instance immediately starts receiving traffic once it is registered to Service Connect / CloudMap. However, it still take another 2 minutes or so for the container to be ready to accept traffic due to its boot up and initialization needs.

I see that CloudMap allows for a Health Check on the Service (ServiceDiscovery.Service.HealthCheckConfig). But the configuration for the ServiceConnectConfiguration.Services property does not allow the definition of a Health Check. I also do not see a way to create the Service directly in CloudMap (ServiceDiscovery::Service) so that I can enable the health check, then map that into the Service Connect settings of the ECS Service defintion.

So how can I enable the CloudMap Service Health Check in combination with the ECS Service "Service Connect" configuration? Or is there another / better way to enforce the not routing traffic to the container instance until it is ready instead of immediately upon registration?

Accepted Answer

After a lot of research and trial/error, I have determined that you can NOT use Cloud Map Service health checks combined with ECS Service Connect. By using a container health check with the Cloud Map Service, the new container instance in a Cloud Map service would be unhealthy until it passes the Container Health Check and therefore it cannot receive traffic until it is ready. I have proven that this part works, but you lose ECS Service Connect discoverability / naming.

Here are some references I found for this:
- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html (see "Service discovery considerations" section and bold "HealthCheckCustomConfig" item toward bottom of list.)
- https://docs.aws.amazon.com/cloud-map/latest/api/API_HealthCheckCustomConfig.html (describes how Custom Health Check will be leveraged)

The problem specifically lies in the DiscoveryName property (AWS::ECS::Service ServiceConnectConfiguration.Services.DiscoveryName). Per https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-service-serviceconnectservice.html,  "The discoveryName is the name of the new AWS Cloud Map service that Amazon ECS creates for this Amazon ECS service". This is the key, because CloudFormation always tries to CREATE the service and fails if the name already exists. For this to work, the CloudFormation (or API/SDK) build/deployment logic would have to change to only create the service name if it doesn't exist - otherwise link/refer to the existing Cloud Map Service. The alternative is to allow the definition of the Health Check in the ServiceConnectConfiguration.Services object.

If you could just link to an existing Cloud Map Service, then the steps below would work (BUT CURRENTLY DO NOT):
- Create the Cloud Map Namespace (AWS::ServiceDiscovery::HttpNamespace)
- Create Cloud Map Service (AWS::ServiceDiscovery::Service) -> specify a HealthCheckCustomConfig containing FailureThreshold only
- Create a Task Definition (AWS::ECS::TaskDefinition) -> add Container HealthCheck object in ContainerDefinitions (Health Check details are specific to your particular needs)
- Create ECS Service (AWS::ECS::Service) that uses Task Definition from above -> add ServiceConnectConfiguration object and provide DiscoveryName = Name or ARN of Cloud Map Service you created above

If you define the Health Check in the ServiceConnectConfiguration.Services object, then the single step below would work :
- Create ECS Service (AWS::ECS::Service) -> add ServiceConnectConfiguration object and provide NameSpace, DiscoveryName and specify a Health Check Config

If either of these (or both) were to be implemented, we could stop directing traffic to instances that are not ready in ECS Service Connect.

Answer

ECS service registration with AWS Cloud Map doesn't support health checks by default. However, you can use Application Load Balancer (ALB) in combination with ECS Service to ensure that the traffic is only routed to the healthy containers.

the official documentation

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html

Here you can find how to

- Create an Application Load Balancer (ALB). You can do this using the AWS::ElasticLoadBalancingV2::LoadBalancer resource in your CloudFormation template.
- Configure the ALB target group with the appropriate health check settings. Use the AWS::ElasticLoadBalancingV2::TargetGroup resource in your CloudFormation template.
- Set the ECS service's loadBalancers property to associate the ECS service with the ALB target group. In this way, ECS will only route traffic to the healthy tasks behind the ALB.
- Update your ECS service configuration in the CloudFormation template to use the ALB. You need to add the loadBalancers property to your ECS service configuration:

```
MyECSService:
  Type: AWS::ECS::Service
  Properties:
    ...
    LoadBalancers:
      - TargetGroupArn: !Ref MyTargetGroup
        ContainerName: MyContainer
        ContainerPort: 80
    ...

```

- Ensure that your tasks have a health check configured. You can use the HEALTHCHECK command in your Dockerfile or configure the health check in your task definition.

By using ALB with ECS, the traffic will only be routed to the healthy containers, as the ALB will perform the health checks and only route traffic to the targets that are deemed healthy.

ECS Service Connect Health Check

Relevant content