By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Creating a EC2-based service in ECS reports deployment failed with no reason - and keeps retrying

0

I am trying to get an ECS deployment using EC2 (not FARGATE) to work. I have gotten a FARGATE version to work.

  1. The task definition is using the public gallery nginx as the container. The task definition (and cluster) are set to EC2
  2. I have two security groups. One routes traffic from the internet to the load balancer; the second routes traffic from the load balancer to the container
  3. I create the service for the cluster and pick the task
  4. For the Network, I choose the same VPC I use in the successful FARGATE deployment and use the ALB to container security group
  5. For the load balancer I choose ALB and set the name of the target group
  6. I start the service

Now there is a bug in the ECS console that does not allow you to set the security group of the ALB. So once the ALB gets created, I edit the security group to set it to the internet to ALB case.

The message 'nginx-ec2-service deployment is in progress. It takes a few minutes' spins for about 40 minutes before the 'Deployments current state' reports '1 Failed, 0 completed' and keeps spinning and eventually it will report 2 failed and keep spinning, etc. This can take hours.

There is no means for me to find out why it failed. All I get is that it failed. I have tried for many days using perturbations to the same result.

Anybody have any idea what could be wrong? How I can get information about WHY it failed? The latter would be GREAT!!!! The FARGATE version works --- why is it SOOO difficult to migrate to EC2 (which is what I want)

Eventually I want an HTTPS on the internet side and port 8080 on the container side. This will cause a great deal more pain.

Topics
ContainersNetworking & Content Delivery
Tags
Amazon Elastic Container ServiceElastic Load BalancingContainersSecurity Group
Language
English
brianR
asked a year ago409 views
2 Answers
1

Hi.

  • Can you confirm containers in task launched successfuly from container logs?
  • What network mode are you using?EC2's default is bridge, so different from awsvpc used in AWS Fargate.
profile picture
EXPERT
Hiroki Takahashi
answered a year ago
  • brianR
    a year ago

    There are no logs, so the container never deployed. I am using awsvpc. I am not sure from the descriptions of these network modes what the consequences of the choice mean, or what the side effects are for other configurations when one chooses one or the other. The only real clear explanations I ever find are in You Tube tutorials if you are lucky enough to find one that covers your problem..

  • Hiroki Takahashi EXPERT
    a year ago

    It seems containers not up and running. Is it possible to pull container image and run on EC2 instance which used from ECS manually?

  • brianR
    a year ago

    I am not quite sure what you mean - create an EC2 instance and run the container on it? That seems like it would be quite a bit of work as I would need a server to handle internet facing requests. I know the container is okay, as I have managed to deploy both nginx AND my desired code (which is a task consisting of two containers) using Fargate. I don't recall at this time if I succeeded in getting my desired code to work using https with Fargate (I did get the nginx working with https and Fargate). The problem with the desired code is that is uses 8080 and AWS makes it VERY hard to change that port on its target groups and security groups. There is often chicken-and-egg problems doing that.

  • Hiroki Takahashi EXPERT
    a year ago

    I think the container image is fine since it is running on Fargate, but I am concerned that the EC2 instance (and its network) that is trying to run the ECS task is properly configured. You say that there are no logs of containers being started, but I thought it would be a good idea to do a docker pull of the container image and see if it can be started with docker exec in order to isolate if it is a configuration issue on the ECS side.

0

Hello Brian,

Could you please check the following troubleshooting steps and let me know if these help you?

  1. Please make sure you have allocated an IAM role to your launch template. Check the documentation for the IAM role here
  2. Please make sure you are using ECS optimised AMI or have the required tooling installed in your AMI.
  3. Please make sure that the autoscaling group you created has "maximum" instance configured for the group to be greater than 0.
  4. Please make sure the instances have either public IP or route to NAT gateway or VPC endpoint so that they can contact Amazon ECS service endpoint.
  5. . If you have more than 1 cluster in your region, please make sure in your launch template you have the following user data configured:
#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config

This documentation here contains all of the above steps with a few more that are needed to configure EC2 instances in ECS, please have a look in case the above top 5 suspects does not work for you.

To troubleshoot further, you could also look login to your instance and check the logs under /var/log/ecs/ecs-agent/*

Thanks, Manish

profile picture
Manish
answered a year ago
  • brianR
    a year ago

    Maybe there is a big piece of information that I have not understood when using ECS with EC2 versus Fargate. Do I have to create an EC2 instance first, before I even LOOK at creating ECS clusters, services, security groups, load balancers, and target groups? I have made the assumption that EC2 instances would get created on demand.

    If that is so, I have created an EC2 instance according to the documentation linked above. Now what do I do when I create clusters, etc. on the ECS page?

  • brianR
    a year ago

    Manish: I created an EC2 instance according to this: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LaunchInstances: I got a 'default' cluster on my ECS page. I do not know what to do with it. It has forced options I don't want when trying to create a service. I have an EC2 instance running. How do I connect that EC2 instance to MY cluster that I have created? How do I attach a certificate? Do I need a load balancer or does the EC2 instance security group take care of HTTPS to HTTP in the container?

  • brianR
    a year ago

    TO add further, a video on how to use ECS with EC2 for an https remote client but http container would be great. Would save me (now) three weeks of frustration.

  • Manish
    a year ago

    I would recommend that you use capacity providers for using EC2 instances with your existing ECS cluster. You will have to create a launch template and autoscaling group beforehand. Please follow this documentation - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-capacity-provider-console-v2.html

    When you are creating the autoscaling group for your capacity provider, please make sure you are configuring the below 2 things in your launch template:

    1. You are using an ECS optimised AMI(you can get one by AWS from marketplace).
    2. Allocated an IAM role to your launch template (This will be under the Advanced settings in Launch template). https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html

    and also in your auto-scaling group you have maximum instances set as more than 0.

    If you follow the above steps, when you launch your service next, you would see in your cluster, under infrastructure, you have container instances getting registered where ECS would then place the service/tasks.

    Generic documentation for how to create launch template can be found here: https://docs.aws.amazon.com/autoscaling/ec2/userguide/create-launch-template.html#create-launch-template-for-auto-scaling You can keep everything as default apart from the AMI, IAM role and your security group configurations for your use-case.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content