Attaching Available EIPs to Auto Scaling Groups for Consistent IP Addresses

Introduction:

Many organizations have strict security requirements, such as allowlisting only specific IP addresses or Elastic IPs (EIPs) on their on-premises firewalls or other resources. However, when using AWS Auto Scaling Groups (ASGs) to automatically manage the scaling of EC2 instances, ensuring that each instance uses an allowlisted IP can be challenging.

This article provides a solution to automatically assign available EIPs to new instances launched by an ASG, ensuring that all instances use a known, allowlisted IP—even as the ASG scales in or out.

The Challenge

Auto Scaling Groups (ASGs) dynamically adjust the number of EC2 instances based on demand, ensuring that your application can handle varying levels of traffic. By default, each new instance launched by an ASG is assigned a dynamic public IP address, which can be problematic if your security policy requires allowlisting specific IPs.

To solve this, we can use Elastic IPs (EIPs), which are static public IP addresses that you can remap between instances. By assigning EIPs to instances launched by an ASG, you ensure that each instance uses an allowlisted IP, satisfying security requirements while maintaining the flexibility of auto-scaling.

The Solution

To achieve this, we utilize a combination of AWS services:

• Auto Scaling Groups (ASG): Automatically manage the number of EC2 instances in your fleet based on demand.
• Elastic IPs (EIPs): A pool of static IP addresses that can be reassigned to any instance in your AWS account.
• AWS Lambda: A serverless compute service that automates the assignment of EIPs to newly launched instances.
• Amazon SNS (Simple Notification Service): A messaging service that triggers the Lambda function when a new instance is launched.
• Lifecycle Hooks: A feature of Auto Scaling that pauses the instance launch process, allowing custom actions to be taken before the instance becomes active.

Step-by-Step Process

1. Assign IAM Roles

Before setting up the other components, it's important to ensure that the necessary permissions are in place. This step involves creating IAM roles:

• SNS Publisher Role: This role allows the Auto Scaling Group to send notifications to the SNS topic.
• Lambda Execution Role: This role grants the Lambda function permissions to interact with EC2 and Auto Scaling services.

2. Create the SNS Topic

The SNS topic acts as a communication hub, receiving notifications from the lifecycle hook and triggering the Lambda function. This decouples the lifecycle event from the Lambda function, allowing for a clean and manageable architecture.

3. Set Up the Auto Scaling Group with a Lifecycle Hook

Next, configure your Auto Scaling Group with a lifecycle hook. A lifecycle hook pauses the launch process of an instance, allowing you to perform custom actions—such as assigning an EIP—before the instance is put into service. When an instance is launched, the lifecycle hook sends a notification to the SNS topic.

4. Configure the Lambda Function

The Lambda function is the core component of this solution. It listens for notifications from the SNS topic and, when triggered, performs the following steps:

1. Check for Existing EIP: The Lambda function first checks whether the instance already has an EIP assigned.
2. Assign an Available EIP: If the instance does not have an EIP, the function searches your EIP pool for an available address and assigns it to the instance.
3. Verify the Assignment: After a short delay, the function verifies that the EIP has been correctly associated with the instance.
4. Retry Logic: If the EIP assignment fails or the verification does not succeed, the Lambda function retries the process several times.
5. Completion: If the function successfully assigns and verifies the EIP, it signals the lifecycle hook to proceed and complete the instance launch. If all retries fail, the function signals the lifecycle hook to abandon the instance launch.

5. Deploy the CloudFormation Template

Finally, automate the entire setup using a CloudFormation template. The CloudFormation template below defines all necessary resources, including the ASG, lifecycle hook, SNS topic, Lambda function, and IAM roles. Deploying the template sets up the entire process, ensuring that each instance launched by your ASG automatically receives an EIP.

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  AutoScalingGroupName:
    Type: String
    Description: Please enter an existing Auto Scaling Group Name

Resources:
  EIPAssignmentSNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: EIPAssignmentTopic

  AutoScalingNotificationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - autoscaling.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: SNSPublishPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - sns:Publish
                Resource: 
                  Ref: EIPAssignmentSNSTopic

  EIPAssignmentLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: EIPAssignmentLambdaPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - ec2:DescribeInstances
                  - ec2:DescribeAddresses
                  - ec2:AssociateAddress
                  - autoscaling:CompleteLifecycleAction
                Resource: "*"
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: "*"

  EIPAssignmentLifecycleHook:
    Type: AWS::AutoScaling::LifecycleHook
    Properties:
      AutoScalingGroupName: 
        Ref: AutoScalingGroupName
      DefaultResult: ABANDON
      HeartbeatTimeout: 180  
      LifecycleHookName: EIPAssignment-Hook
      LifecycleTransition: autoscaling:EC2_INSTANCE_LAUNCHING
      NotificationTargetARN: 
        Ref: EIPAssignmentSNSTopic
      RoleARN: 
        Fn::GetAtt: 
          - AutoScalingNotificationRole
          - Arn

  EIPAssignmentLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: EIPAssignmentFunction
      Code:
        ZipFile: |
          const { EC2Client, DescribeInstancesCommand, DescribeAddressesCommand, AssociateAddressCommand } = require("@aws-sdk/client-ec2");
          const { AutoScalingClient, CompleteLifecycleActionCommand } = require("@aws-sdk/client-auto-scaling");

          async function checkEIP(instanceId) {
              const ec2Client = new EC2Client({});
              const params = { InstanceIds: [instanceId] };
              const describeInstancesResponse = await ec2Client.send(new DescribeInstancesCommand(params));
              const instance = describeInstancesResponse.Reservations[0].Instances[0];

              // Get a list of all EIPs in your account
              const describeAddressesResponse = await ec2Client.send(new DescribeAddressesCommand({ Filters: [{ Name: 'domain', Values: ['vpc'] }] }));
              const eipSet = new Set(describeAddressesResponse.Addresses.map(addr => addr.PublicIp));

              // Check if the instance's public IP is one of your EIPs
              for (const eni of instance.NetworkInterfaces) {
                  if (eni.Association && eipSet.has(eni.Association.PublicIp)) {
                      console.log("Instance already has EIP:", eni.Association.PublicIp);
                      return eni.Association.PublicIp;
                  }
              }

              return null;
          }

          async function associateEIP(instanceId) {
              const ec2Client = new EC2Client({});
              // Find an available EIP
              const describeAddressesResponse = await ec2Client.send(new DescribeAddressesCommand({ Filters: [{ Name: 'domain', Values: ['vpc'] }] }));
              const availableAddress = describeAddressesResponse.Addresses.find(addr => !addr.AssociationId);

              if (!availableAddress) {
                  throw new Error("No available EIP found");
              }

              // Associate the EIP with the instance
              const associateParams = {
                  AllocationId: availableAddress.AllocationId,
                  InstanceId: instanceId
              };
              await ec2Client.send(new AssociateAddressCommand(associateParams));
              console.log("Successfully associated EIP", availableAddress.PublicIp, "with instance", instanceId);
              return availableAddress.PublicIp;
          }

          function randomDelay() {
              const min = 2000;
              const max = 4000;
              const delay = Math.floor(Math.random() * (max - min + 1)) + min;
              return new Promise(resolve => setTimeout(resolve, delay));
          }

          exports.handler = async (event) => {
              const asClient = new AutoScalingClient({});
              const instanceId = JSON.parse(event.Records[0].Sns.Message).EC2InstanceId;
              let success = false;
              let retries = 10;  // Increased retries

              while (!success && retries > 0) {
                  try {
                      retries--;

                      // Step 1: Check if the instance already has an EIP from your pool
                      let assignedEIP = await checkEIP(instanceId);
                      if (!assignedEIP) {
                          // Step 2: Associate an EIP
                          assignedEIP = await associateEIP(instanceId);
                      }

                      // Random delay between 2 and 4 seconds before verification
                      await randomDelay();

                      // Step 3: Verify the association
                      const verifiedEIP = await checkEIP(instanceId);
                      if (verifiedEIP === assignedEIP) {
                          console.log("Successfully verified EIP:", verifiedEIP);
                          success = true;
                      } else {
                          console.error("Verification failed. Retrying...");
                      }
                  } catch (error) {
                      console.error("Error during EIP assignment:", error.message);
                      if (retries === 0) {
                          console.error("Max retries reached. Failing the lifecycle action.");
                          const message = JSON.parse(event.Records[0].Sns.Message);
                          const lifecycleParams = {
                              AutoScalingGroupName: message.AutoScalingGroupName,
                              LifecycleHookName: message.LifecycleHookName,
                              LifecycleActionToken: message.LifecycleActionToken,
                              LifecycleActionResult: "ABANDON"
                          };
                          await asClient.send(new CompleteLifecycleActionCommand(lifecycleParams));
                          return;
                      }
                  }
              }

              if (success) {
                  const message = JSON.parse(event.Records[0].Sns.Message);
                  const lifecycleParams = {
                      AutoScalingGroupName: message.AutoScalingGroupName,
                      LifecycleHookName: message.LifecycleHookName,
                      LifecycleActionToken: message.LifecycleActionToken,
                      LifecycleActionResult: "CONTINUE"
                  };
                  await asClient.send(new CompleteLifecycleActionCommand(lifecycleParams));
              }
          };
      Handler: index.handler
      Runtime: nodejs20.x  
      Role: 
        Fn::GetAtt: 
          - EIPAssignmentLambdaRole
          - Arn
      Timeout: 60

  LambdaSNSSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: 
        Fn::GetAtt: 
          - EIPAssignmentLambdaFunction
          - Arn
      Protocol: lambda
      TopicArn: 
        Ref: EIPAssignmentSNSTopic

  LambdaSNSInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: 
        Fn::GetAtt: 
          - EIPAssignmentLambdaFunction
          - Arn
      Action: lambda:InvokeFunction
      Principal: sns.amazonaws.com
      SourceArn: 
        Ref: EIPAssignmentSNSTopic

Outputs:
  EIPAssignmentLambdaFunction:
    Value: 
      Ref: EIPAssignmentLambdaFunction

*Please make sure your account in that region has enough EIPs, otherwise ASG can’t launch instance successfully.

Benefits of This Approach

• Consistent IP Allowlisting: By using a pool of EIPs, your instances consistently have known IP addresses, simplifying firewall rules and security configurations.
• Automation: The process is fully automated using AWS Lambda and CloudFormation, reducing the need for manual intervention and minimizing the risk of errors.
• Scalability: This solution works seamlessly with Auto Scaling Groups, ensuring that your IP allowlisting requirements are met even as your infrastructure scales dynamically.

Conclusion

Assigning fixed EIPs to instances in an Auto Scaling Group allows you to maintain consistent IP addresses for allowlisting, even in dynamic cloud environments. By leveraging AWS services such as Lambda, SNS, and lifecycle hooks, you can automate the process, ensuring that each new instance launched by your ASG is assigned a known and allowlisted IP address.

This solution offers a robust, scalable, and automated way to meet strict security requirements while enjoying the flexibility and power of Auto Scaling Groups. With the provided CloudFormation template, you can deploy this solution in your AWS environment within minutes, ensuring that your Auto Scaling instances always use the IP addresses you need.

Next Steps

• Deploy the provided CloudFormation template in your AWS environment.
• Test the solution by launching new instances in your Auto Scaling Group and verifying that they receive the correct EIPs.
• Adjust the Lambda function logic or the EIP pool as needed to suit your specific use case.

By following these steps, you can ensure that your Auto Scaling Group instances always have the IP addresses required for your security and operational needs.

Disclaimer:

This script has been tested multiple times in a controlled lab environment and has consistently demonstrated satisfactory performance. However, it is provided on an "as-is" basis, and while we have made every effort to ensure its reliability, we cannot guarantee its accuracy or functionality in other environments. It is the user's responsibility to thoroughly evaluate and test this script in a non-production environment before deploying it in production. Use of this script is at your own risk, and AWS assumes no liability for any damages or issues arising from its use. For assistance or inquiries, please contact AWS support team.