Elastic Beanstalk´s ASG cannot create EC2 instances

Question

**The problem**

I´m using Elastic Beanstalk to deploy applications alongside GitHub Actions, when the action gets activated, Beanstalk creates an ASG where the desired capacity creates at least 1 instance with the containerized application.

For some reason, the ASG provided by Beanstalk started to set as unhealthy the instances almost in an immediate way and terminates them. This process repeats 5 or 6 times and then returns an error state to the beanstalk application.

The ASG remains in *Provisioning state* and when I looked at the ASG activity history log I got the following:

| Status | Description | Cause |
| --- | --- | --- |
| Cancelled | Launching a new  EC2 instance. Status Reason: Instance Became unhealthy while waiting for instance to be in inService state. Termination Reason: Client.InternalError: Client error on launch | At 2022-01-13 an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1 |

And the EB environment events  throw 4 errors:

1. Creating Auto Scaling group named: awseb-[...]-AWSEBAutoScalingGroup-1V0R8Z9EJ8G8J failed. Reason: Group did not stabilize. {current/minSize/maxSize} group size = {0/1/1}.
2. Service:AmazonCloudFormation, Message:Stack named 'awseb-e-rvjtnttttf-immutable-stack' aborted operation. Current state: 'CREATE_FAILED' Reason: The following resource(s) failed to create: [AWSEBAutoScalingGroup].
3. Failed to deploy application.
4. Cannot complete command execution on the following instances as they are no longer running: [i-03449eff8756123c2].

**The following steps have been taken at the moment**

1. Review of IAM role permissions to allow creating EC2 instances.
2. Review of SG and secure connection between the load balancer and target group

**Identified activities before the problem**
1. Enable ELB health check with 300 grace period for two of our ASG

**Personal point of view**

The problem seems to be not directly with Beanstalk but between TG and the instances, maybe a VPC endpoint is needed to return health status from EC2 to TG?

`Client -> LB -> TG -[HERE]> EC2`

Accepted Answer

Thanks to everyone! Your answers helped me a lot

I review the problem described [here](https://aws.amazon.com/es/premiumsupport/knowledge-center/encrypted-volumes-stops-immediately/) and realized that the main problem was actually with the EBS volumes, the EBS encryptions were enabled by default and somehow my current EB applications version cannot support encrypted EBS.

That's where the "client error on launch" comes from.

Answer

When I saw behavior like this, I extended how long the Auto scalinggroup waited after the deploy to check for HealthCheck. in my case, the instances simply needed more time.

in .ebextensions/00_autoscaling.config
```
Resources:
  AWSEBAutoScalingGroup:
    Type: "AWS::AutoScaling::AutoScalingGroup"
    Properties:
      HealthCheckType: ELB
      HealthCheckGracePeriod: 600 # wait 2x the default before checking
```

This doc has a section on HealthCheckGracePeriod
https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html

Also, grab the logs from the server: in some cases the nginx / access logs can help to understand if the healthcheck is reaching your instance, but is returning non 200s.

Answer

Consider temporarily [suspending and resuming a process for an Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html). Be aware it can prevent other processes from functioning properly but it may give you an avenue to investigate what is going on and track down the source of the issue.

Elastic Beanstalk´s ASG cannot create EC2 instances

Relevant content