Hello! We have an AWS solution that has a Global Accelerator with static IPs as the entry point, followed by an Elastic Load Balancer, and then an API Gateway. We did all this just because we needed an API Gateway with static IPs so they could be approved (Enterprise system).
The API Gateway then stores data on an S3 Bucket, which replicates it to an external tenant through a Replication Rule.
After many months of working unattended, the system stopped working on the third week of January 2024. After going to S3, I found that the replication rules were missing (wiped out). I went into CloudTrail and found that on January 12th, the network interfaces were deleted by AWS. (I'm assuming this would be the trigger that caused the replication to be deleted as a cascade effect, there are no logs in Cloud Trail related to S3 or Replication Rules)
We are trying to find what caused this. Could one of the services that we are using have been deprecated? What could have happened?
Thanks!
Here a log example of one of the Network Interface deletions from Cloud Trail:
(I apologize, for some reason in the editor the log appears well idented and with one key/value pair per line, but once is published is not anymore)
{
"eventVersion": "1.09",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROXXXXXXXXXXXXX:ElasticLoadBalancing",
"arn": "arn:aws:sts::4602XXXXXXXXX:assumed-role/AWSServiceRoleForElasticLoadBalancing/ElasticLoadBalancing",
"accountId": "460XXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAXXXXXXXXXXXXX",
"arn": "arn:aws:iam::460XXXXXXXXXXX:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing",
"accountId": "460XXXXXXXXXXX",
"userName": "AWSServiceRoleForElasticLoadBalancing"
},
"attributes": {
"creationDate": "2024-01-12T23:56:25Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "elasticloadbalancing.amazonaws.com"
},
"eventTime": "2024-01-12T23:56:26Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "DeleteNetworkInterface",
"awsRegion": "us-east-2",
"sourceIPAddress": "elasticloadbalancing.amazonaws.com",
"userAgent": "elasticloadbalancing.amazonaws.com",
"requestParameters": {
"networkInterfaceId": "eni-0818b7XXXXXXXXXX"
},
"responseElements": {
"requestId": "a9d0a2ca-67c9-XXXXXXXXXXX",
"_return": true
},
"requestID": "a9d0a2ca-67c9-4de0-XXXXXXXXXXX",
"eventID": "ae21b158-2a35-45a4-XXXXXXXXXXX",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "46025XXXXXXXXXXX",
"eventCategory": "Management"
}
Thank you. After yesterday's internal investigation, as you pointed out, we believe there is no correlation between the Network Interfaces being deleted by the ELB on the 12th and the S3 Replication Rules disappearing without any event logged in CloudTrail around the 19th.
Thanks again.