How do I terminate or rebuild my AWS Elastic Beanstalk environment when the AWSEBSecurityGroup fails to delete?

Short description

AWSEBSecurityGroup is an Amazon Elastic Compute Cloud (Amazon EC2) security group created by the underlying AWS CloudFormation stack through Elastic Beanstalk. If AWSEBSecurityGroup is used by an elastic network interface (ENI) or security groups created outside your Elastic Beanstalk environment, terminating or rebuilding the environment can fail and return an error.

Note: It's a best practice to manage the resources created by Elastic Beanstalk with the Elastic Beanstalk console, Elastic Beanstalk Command Line Interface (EB CLI), or the AWS Command Line Interface (AWS CLI).

Resolution

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.

To resolve this error, choose one of the following options:

• (Option A) Delete the AWS CloudFormation stack created by the Elastic Beanstalk environment in either the AWS CloudFormation console or the AWS CLI, and then terminate the Elastic Beanstalk environment.
• (Option B) Remove the dependent objects from your Elastic Beanstalk security group, and then terminate the Elastic Beanstalk environment.

Important: Be sure to enter the correct values for your stack name, environment name, AWS Region, and other environment-specific variables in the AWS CLI.

(Option A) Delete the stack from the AWS CloudFormation console

You can delete the stack either from the AWS CloudFormation console or the AWS CLI.

AWS CloudFormation console:

1.    Open the AWS CloudFormation console.

2.    From the Stack Name column, choose the stack that failed to delete.

Note: The Status column of your stack shows DELETE_FAILED.

3.    From the Actions menu, choose Delete Stack.

4.    In the Delete Stack pop-up window, choose AWSEBSecurityGroup, and then choose Yes, Delete.

Important: Be sure to delete the security group. The AWSEBSecurityGroup isn't deleted automatically when you delete the stack. If the environment termination fails due to another dependent resource, then choose the appropriate resource in the DELETE_FAILED state from the Delete Stack pop-up window. Then, choose Yes, Delete.

5.    Terminate or rebuild the Elastic Beanstalk environment.

AWS CLI:

1.    To delete the AWS CloudFormation stack, run the following command:

aws cloudformation delete-stack --stack-name awseb-e-ztrauxujck-stack --retain-resources "AWSEBSecurityGroup" --region us-east-1

2.    To terminate the Elastic Beanstalk environment, run the following command:

aws elasticbeanstalk terminate-environment --environment-name my-env --region us-east-1

3.    To rebuild the Elastic Beanstalk environment, run the following command:

aws elasticbeanstalk rebuild-environment --environment-name my-env --region us-east-1

(Option B) Find and remove the dependent objects for the Elastic Beanstalk security group

To find a dependent elastic network interface or a dependent security group that's preventing the AWSEBSecurityGroup from being deleted, complete the following steps with the Amazon EC2 console or the AWS CLI.

Amazon EC2 console:

1.    Get the security group ID of AWSEBSecurityGroup from the error message in the Elastic Beanstalk event stream.

2.    Open the Amazon EC2 console.

3.    In the navigation pane, choose Security Groups, and then choose the AWSEBSecurityGroup ID that you obtained in step 1.

4.    Choose Actions, and then choose Delete Security Group to find the dependencies on AWSEBSecurityGroup.

5.    In the Delete Security Groups pop-up window, choose View your associated instance.
Choose Actions, and then choose Networking.
Choose Change Security Group.
Clear the AWSEBSecurityGroup ID and Assign Security Groups check boxes.

6.    Go back to the Delete Security Groups pop-up window, and then choose View your referencing security groups.
To remove the association of the AWSEBSecurityGroup ID, choose Actions, and then choose Edit inbound rules or Edit outbound rules depending on your requirements.

7.    Terminate or rebuild the Elastic Beanstalk environment.

AWS CLI:

1.    Get the security group ID of AWSEBSecurityGroup from the error message in the Elastic Beanstalk event stream.

2.    Save the following bash script as sg_dependency.sh:

#!/bin/bash
SecurityGroupID=$1
Region=$2
DependentENI=$(aws ec2 describe-network-interfaces --filters Name=group-id,Values=[$SecurityGroupID] --region $Region | jq '.NetworkInterfaces[].NetworkInterfaceId')
DependentSGingress=$(aws ec2 describe-security-groups --filters Name=ip-permission.group-id,Values=[$SecurityGroupID] --region $Region | jq '.SecurityGroups[] .GroupId')
DependentSGegress=$(aws ec2 describe-security-groups --filters Name=egress.ip-permission.group-id,Values=[$SecurityGroupID] --region $Region | jq '.SecurityGroups[] .GroupId')
echo "The dependent ENI is $DependentENI"
echo "The security group id with a dependency in ingress rule : $DependentSGingress"
echo "The security group id with a dependency in egress rule :  $DependentSGegress"

3.    Install the jq utility from the jq website.

4.    To get your elastic network interface ID, security group ID, or both, run the following command:

./sg_dependency.sh sg-111aaa22 us-east-1

Note: Replace sg-111aaa22 with your AWSEBSecurityGroup ID. Replace us-east-1 with your AWS Region.

5.    To find the security groups attached to the elastic network interface ID that you obtained in step 4, run the following command:

aws ec2 describe-network-interface-attribute --network-interface-id eni-1099d901 --attribute groupSet

6.    To detach the AWSEBSecurityGroup ID, run the following command.

Important: Be sure to exclude the AWSEBSecurityGroup ID in this command, and include other security group IDs to remain attached to the network-interface.

aws ec2 modify-network-interface-attribute --network-interface-id eni-1099d901 --groups sg-030644d6c95b6470a sg-d34a33a8 sg-059c081ab30c3e38e sg-0e4eabc747368e6c9

Important: If the command output returns the "An error occurred (AuthFailure) when calling the ModifyNetworkInterfaceAttribute operation: You don't have permission to access the specified resource" error, remove AWSEBSecurityGroup from the AWS service that created that elastic network interface.

7.    Run the describe-security-groups command to list all the ingress and egress rules of the dependent security group that you obtained in step 4.

8.    Run the revoke-security-group-ingress or revoke-security-group-egress command based on the output from step 7.

For example, if the AWSEBSecurityGroup ID is sg-111aaa22 and the output from step 8 has that security group ID in ingress rules, run the following command:

aws ec2 revoke-security-group-ingress --group-id sg-0018a2fbdcd5797ea --ip-permissions '[{"PrefixListIds":[],"FromPort":0,"IpRanges":[],"ToPort":65535,"IpProtocol":"tcp","UserIdGroupPairs":[{"UserId":"123456789102","GroupId":"sg-111aaa22"}],"Ipv6Ranges":[]}]'

Note: The UserId is the AWS account ID.

9.    To terminate the Elastic Beanstalk environment, run the following command:

aws elasticbeanstalk terminate-environment --environment-name my-env --region us-east-1

10.    To rebuild the Elastic Beanstalk environment, run the following command:

aws elasticbeanstalk rebuild-environment --environment-name my-env --region us-east-1

Related information

Troubleshooting (Elastic Beanstalk)