Network Interfaces were deleted by AWS on Jan 12th 2024

Question

Hello! We have an AWS solution that has a Global Accelerator with static IPs as the entry point, followed by an Elastic Load Balancer, and then an API Gateway. We did all this just because we needed an API Gateway with static IPs so they could be approved (Enterprise system).

The API Gateway then stores data on an S3 Bucket, which replicates it to an external tenant through a Replication Rule.

After many months of working unattended, the system stopped working on the third week of January 2024. After going to S3, I found that the replication rules were missing (wiped out). I went into CloudTrail and found that on January 12th, the network interfaces were deleted by AWS. (I'm assuming this would be the trigger that caused the replication to be deleted as a cascade effect, there are no logs in Cloud Trail related to S3 or Replication Rules)

We are trying to find what caused this. Could one of the services that we are using have been deprecated? What could have happened?

Thanks!

Here a log example of one of the Network Interface deletions from Cloud Trail:

(I apologize, for some reason in the editor the log appears well idented and with one key/value pair per line, but once is published is not anymore)

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROXXXXXXXXXXXXX:ElasticLoadBalancing",
        "arn": "arn:aws:sts::4602XXXXXXXXX:assumed-role/AWSServiceRoleForElasticLoadBalancing/ElasticLoadBalancing",
        "accountId": "460XXXXXXXXXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROAXXXXXXXXXXXXX",
                "arn": "arn:aws:iam::460XXXXXXXXXXX:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing",
                "accountId": "460XXXXXXXXXXX",
                "userName": "AWSServiceRoleForElasticLoadBalancing"
            },
            "attributes": {
                "creationDate": "2024-01-12T23:56:25Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "elasticloadbalancing.amazonaws.com"
    },
    "eventTime": "2024-01-12T23:56:26Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "DeleteNetworkInterface",
    "awsRegion": "us-east-2",
    "sourceIPAddress": "elasticloadbalancing.amazonaws.com",
    "userAgent": "elasticloadbalancing.amazonaws.com",
    "requestParameters": {
        "networkInterfaceId": "eni-0818b7XXXXXXXXXX"
    },
    "responseElements": {
        "requestId": "a9d0a2ca-67c9-XXXXXXXXXXX",
        "_return": true
    },
    "requestID": "a9d0a2ca-67c9-4de0-XXXXXXXXXXX",
    "eventID": "ae21b158-2a35-45a4-XXXXXXXXXXX",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "46025XXXXXXXXXXX",
    "eventCategory": "Management"
}

Accepted Answer

From your description, I guess you may refer to [1] this blog's approach. The concatenation architecture may be as follows:

Client(Enterprise) <---> (Static IP) Global Accelerator <---> ALB <---> Private APIGW <---> S3

Judging from your CloudTrail, ALB is an elastically scalable load balance service, so the underlayer of ALB is composed of multiple Managed Elastic Network Interfaces (ENI). However, these ENIs may be replaced due to various reason like, software updates, failover or autoscaling of ALB. It is possible that the ENIs will be created or deleted by AWSServiceRoleForElasticLoadBalancing. When ALB performs these operations, it will provide graceful period and draining mechanisms to ensure that production services will not be affected when ENI is replaced.

It is recommended that you confirm whether your service exception is directly related to this event. If you have individual resource issues, it is still recommended that you contact AW technical support to further troubleshoot your resources.

[1] https://aws.amazon.com/tw/blogs/networking-and-content-delivery/accessing-an-aws-api-gateway-via-static-ip-addresses-provided-by-aws-global-accelerator/

Network Interfaces were deleted by AWS on Jan 12th 2024

Relevant content