App Mesh - ECS Tasks Failing Health Check After Adding Envoy Proxy to Task Definition

Hey everyone,

I currently have a application that consists of three services in a ECS Fargate cluster. I also have a ALB which routes external traffic to these services as well. These three services talk also talk internally to each other which is what I would like to enclose in a App Mesh. I set up AWS Cloud Map for my three services and created the corresponding mesh/virtual services/virtual nodes. The App mesh leverages AWS Cloud Map for service discovery. All of my tasks were running healthy and AWS Cloud Map picked up all the service instances correctly.

My issue begins when I enable app mesh integration in the ECS task definitions/adding the envoy proxy container to the task definition. When I do this my ECS tasks begin to create and then automatically delete themselves due to the ALB health checks failing against my tasks.

Basically when I just have a single container in my task which holds my service, the ALB health check passes. When I add the envoy proxy sidecar container to my task, the envoy container returns healthy but my service container returns unhealthy until the task deletes itself.

Specifics

ALB forwards requests to target groups via HTTPS protocol on port 7001.

ALB health checks use HTTPS on port 7001. It checks every 30 seconds and will timeout after 5 seconds. It will turn unhealthy after 3 consecutive failures.

Task definition for example service has two containers. One for the service itself and the other as the envoy proxy sidecar.

Container holding the service itself has Host and Container port mappings of port 7001.

Containing holding the service itself has a dependency that the envoy proxy container is healthy before creating container.

ECS task definition has the following proxy configuration values:

Proxy container name: envoy

Type: APPMESH

Ignored UID: 1337

App ports: 7001

Proxy ingress port: 15000

Proxy egress port: 15001

Egress ignored IPs: 169.254.170.2,169.254.169.254

I've spent the past two days trying to figure out why my service containers return unhealthy after adding the envoy proxy container and I've exhausted all my options, I really need some guidance here and would greatly appreciate it! :)