Service on fargate container won't start if I disable public IP

0

I run a small web app on fargate container. The container by default has a public IP address associated with it. I don't need it to have a public IP.

So I created a new service (exactly like my current one) with the only difference being that I switched off the public IP flag.

I also added endpoints to my VPC for ECR (dkr, api), S3 and SMTP.

The security group has all traffic enabled for both inbound and outbound. I am using the default VPC created by AWS which has only public subnets.

The problem is that the same task won't start on the new service. This same task works fine in my current (old) service which has a public IP.

I get an error: "Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'.

What could I be missing?

shuaybi
asked a month ago152 views
1 Answer
1
Accepted Answer

Hello.

From what I've heard about the situation, I think it's possible that communication to ECR and other services is not working.
To access ECR, the VPC endpoint should have at least the following:
It seems like you have already created one, so I think the VPC endpoint is fine.
https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html

  • com.amazonaws.${AWS::Region}.s3
  • com.amazonaws.${AWS::Region}.ecr.dkr
  • com.amazonaws.${AWS::Region}.ecr.api

The next thing we need to check is the security group configured for the VPC endpoint.
The VPC endpoint security group must allow HTTPS in the inbound rule.
First, please check the security group settings.

It seems like the deployment circuit breaker is running, so you might want to try disabling it once.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html

profile picture
EXPERT
answered a month ago
profile picture
EXPERT
reviewed a month ago
  • Make sure you have added the S3 gateway and not the S3 interface end point. This is assuming your docker image is in ECR.

  • Gary - My docker image is in ECR and I was missing the S3 gateway end point which I added. However it didn't help. I get the same error.

    Riku - My security group has ALL TCP (0-65535, 0.0.0.0/0) Inbound and Outbound rules.

    I am still getting the exact same error:

    Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'."

  • I've seen similar errors occur since the new ECS UI. The cause at that time was that ecsTaskExcutionRole did not have access rights to CloudWatch Logs. Try setting the policy for "logs:CreateLogGroup".

  • Riku - I added the logs policy to ecsTaskExecutionRole. No luck. Like I mentioned, the same service with the exact same task definition works if I just enable the public IP flag. But fails if I turn off the public IP flag.

  • I think there is a problem with the network settings or the VPC endpoint. By the way, have you configured log output settings from Fargate to CloudWatch Logs? If you have done so, please also add the following CloudWatch Logs VPC endpoint.

    com.amazonaws.${AWS::Region}.logs
    

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions