EC2 Security Group: Only works when removed and readded

0

Hi,

I've been trying to deploy an EC2 instance into my VPC, of which I'm confident on the configurations (ingress and egress rules, route tables, etc.). When I deploy the EC2 instance (tested with console, terraform, and cloud formation) with my security group I am unable to access the instance via the publicly assigned IP address. I have run the AWS Network Access Analyzer and Reachability Analyzer and nothing is showing as unreachable or incorrect.

After much trial and error the only solution that I was able to find was to:

  1. remove my security group (sg1)
  2. attach another (sg2)
  3. save
  4. remove sg2
  5. attach sg1 (my desired security group)
  6. save

After this the networking rules behave as expected.

  • Have I missed something here?
  • As I'm primarily using IaC does this have an impact e.g. other resources automatically being created such as Network Interfaces?
  • I have also left some instances running to see if there was some propergation aspect however this doesn't seem to be the case.

I'm a little lost in terms of how I'd deploy the instances without having to manually correct the SG each time!

Below are various deployments I've tested, with XYZ replacing the actual sensitive data.

Terraform code (SG and EC2 instance):

...
### public
resource "aws_security_group" "sg_public" {
  name        = "sg_public"
  description = "Allow ingress from internal IP for HTTP, HTTPS, SSH. Allow egress HTTP, HTTPS for external api calls"
  vpc_id      = local.id_vpc

  tags = {
    Name = "sg_public"
  }

}
...
# also tested with aws_security_group_rule
resource "aws_vpc_security_group_egress_rule" "sg_public_rule_egress_allow_HTTP" {
  security_group_id = aws_security_group.sg_public.id
  cidr_ipv4         = "XYZ/0"
  ip_protocol       = "tcp"
  from_port         = 80
  to_port           = 80
}
...
resource "aws_instance" "ec2_small_test" {

  ami                         = "ami-0f71013b2c8bd2c29" # Amazon Linux 2023 AMI
  instance_type               = "t2.micro"
  vpc_security_group_ids      = [aws_security_group.sg_public.id]
  subnet_id                   = local.id_subnet
  associate_public_ip_address = true
  user_data                   = file("webapp.sh") # simple "hello world"

}
...

CF template that I tested deploying the SG with:

Description: CloudFormation template for an SG

Parameters:
  myVPC:
    Type: String
    Default: vpc-XYZ
    Description: vpc id

Resources:
  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: TestFromCF
      GroupDescription: Allow http to client host
      VpcId: !Ref myVPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: XYZ/32
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0

CF template that I tested to launch the instance:

Description: CloudFormation template for an EC2 instance referencing an existing
  security group.

Parameters:
  SubnetId:
    Type: String
    Default: subnet-XYZ
    Description: The ID of the subnet to launch the instance in.
  SecurityGroupId:
    Type: String
    Default: sg-XYZ
    Description: The ID of the existing security group.

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-0f71013b2c8bd2c29
      SubnetId: !Ref SubnetId
      SecurityGroupIds:
        - !Ref SecurityGroupId
      UserData: !Base64 |
        #!/bin/bash
        # Contents of webapp.sh
        echo "Hello, World!" > /var/www/html/index.html
        yum update -y
        yum install -y httpd
        systemctl start httpd
        systemctl enable httpd
      Tags:
        - Key: Name
          Value: hello_world_cf

Outputs:
  InstanceId:
    Description: The ID of the created EC2 instance.
    Value: !Ref EC2Instance
  InstancePublicIp:
    Description: The public IP address of the EC2 instance.
    Value: !GetAtt EC2Instance.PublicIp
3 Answers
0

Detaching and re-attaching the security group manually should not be required. There is likely a configuration issue that is not immediately obvious such as a similarly named SG without the required access. I noticed in the Terraform code provided you are using aws_vpc_security_group_egress_rule allow access to port 80 on the instance. This conflicts with the CloudFormation you posted and is likely not what you want. Instead use aws_vpc_security_group_ingress_rule. I recommend trying to deploy with the correct ingress rule using Terraform and see if that resolves your issue.

profile pictureAWS
answered 2 months ago
  • Hi Sean,

    That was my bad, I accidentally copied the wrong section over in the post.

    I've been using aws_vpc_security_group_ingress_rule and the issue still persists.

    Are there any CF templates that provide a minimal VPC config and launching an instance? I'm trying to rule out if it's a IaC (CF or TF) or if it's the ordering of when public IPs are assigned by the network interface.

0

Based on the information you've provided, it seems you're encountering an unusual issue with EC2 security groups. Here are some insights and potential explanations for the behavior you're experiencing:

  1. Security Group Rule Propagation: While security group changes typically apply immediately, there can sometimes be a brief delay in rule propagation. However, this delay is usually very short and shouldn't require the manual removal and re-addition of security groups.

  2. Network Interface Association: When you launch an EC2 instance, a network interface is automatically created and associated with it. This network interface is linked to the security group(s) you specify. The behavior you're seeing could potentially be related to how this association is being made or updated.

  3. Infrastructure as Code (IaC) Impact: Using IaC tools like Terraform or CloudFormation shouldn't inherently cause this issue. These tools interact with the AWS API in a similar way to the console, so the behavior should be consistent.

  4. Security Group Configuration: Ensure that your security group rules are correctly configured. For a web server, you typically need:

    • Inbound rule allowing HTTP (port 80) from your desired source IP range
    • Inbound rule allowing HTTPS (port 443) if you're using SSL/TLS
    • Inbound rule allowing SSH (port 22) for management access
    • Outbound rules are typically allowed to all destinations by default
  5. VPC and Subnet Configuration: Verify that your VPC and subnet configurations are correct, including route tables and internet gateway associations for public subnets.

  6. Instance Public IP: Confirm that your instance is indeed receiving a public IP address when launched.

To address this issue and avoid manual intervention:

  1. Double-check all your network configurations, including VPC, subnet, route tables, and internet gateway settings.

  2. Verify that your security group rules are correctly defined in your IaC code.

  3. Consider creating a new security group from scratch and applying it to a test instance to see if the issue persists.

  4. If the problem continues, you may want to engage AWS support to investigate if there's an underlying issue with your account or the specific VPC/subnet you're using.

  5. As a workaround, you could potentially script the process of detaching and reattaching the security group post-deployment, although this isn't an ideal long-term solution.

Remember, security groups are stateful, so you don't need to explicitly allow outbound traffic for responses to allowed inbound traffic. Also, ensure that your instance's operating system firewall isn't interfering with incoming connections.

If none of these suggestions resolve the issue, it may be worth exploring if there are any account-specific quirks or regional issues by testing in a different AWS region or account.
Sources
Amazon EC2 security groups for your EC2 instances - Amazon Elastic Compute Cloud
AWS::EC2::SecurityGroup - AWS CloudFormation
Change the security groups for your Amazon EC2 instance - Amazon Elastic Compute Cloud
Allow access to EC2 applications with a security group | AWS re:Post

profile picture
answered 2 months ago
0

Hello.

The security group rule settings defined in IaC allow access via HTTP, so I think there is no problem with the settings themselves.
It seems that Apache is installed using user data when creating EC2, but it may take a while for Apache to start after the installation is complete.
In other words, I thought that if I waited a while and then accessed it, I would be able to access it without any problems.

It is also possible to use "cfn-signal" to complete the CloudFormation stack creation upon completion of the user data script.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-signal.html

profile picture
EXPERT
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions