* Found that the duplication event occurred consecutively. * Parsed each batch of 50 audit events fetched from the LookupAPI of CloudTrail and checked the duplication with eventID and found that the duplication event came from AWS itself. * Also collected the audit logs and stored them in the file and found that only for **us-east-1** the audit logs were duplicate. * Created a script to find the duplication from collected logs and also find the same results. * The count of duplicate events differs every time. But whenever I tried to reproduce this bug found that every time the same events fetched were duplicated. * No pattern found for duplicate events except the event ids of the duplicate events are the same every time.

Duplicate events in batch of 50 while fetching audit data through LookupEvents API of CloudTrail.

Found that the duplication event occurred consecutively.
Parsed each batch of 50 audit events fetched from the LookupAPI of CloudTrail and checked the duplication with eventID and found that the duplication event came from AWS itself.
Also collected the audit logs and stored them in the file and found that only for us-east-1 the audit logs were duplicate.
Created a script to find the duplication from collected logs and also find the same results.
The count of duplicate events differs every time. But whenever I tried to reproduce this bug found that every time the same events fetched were duplicated.
No pattern found for duplicate events except the event ids of the duplicate events are the same every time.

Topics

Management & Governance

Tags

AWS CloudTrail

Language

English

Parth Dadhaniya

asked a year ago414 views

1 Answer

Newest
Most votes
Most comments

Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.

Hi There

Do you have multiple CloudTrail Trails configured in different regions? If so, you could be seeing duplicates for global service events. Examples of global service events are AWS IAM, CloudFront, and AWS STS. If these are the types of duplicate events you are seeing, make sure you are not logging "Management Events" in multiple CloudTrails. See https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-concepts.html#cloudtrail-concepts-global-service-events for additional info.

EXPERT

Matt-B

answered a year ago

Parth Dadhaniya
a year ago
Thanks for this information. But I am fetching the audit logs which are generated before CloudTrail is created. For that purpose, I am using the LookupEvents API to fetch that logs. As per my knowledge, the creation of CloudTrail and the duplicate events doesn't relate to each other.
Matt-B EXPERT
a year ago
Can you post an example of a duplicate event?

Parth Dadhaniya

a year ago

Sure.

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "QWERTYUIOPASDFGHJKLZXCV:1cefa620-1234-1234-1234-24bddba12345",
    "arn": "arn:aws:sts::012345678912:assumed-role/test-role/1cefa620-1234-1234-1234-24bddba12345",
    "accountId": "012345678912",
    "accessKeyId": "QWERTYUIOPASDFGHJKLZXC",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "QWERTYUIOPASDFGHJKLZXCV",
        "arn": "arn:aws:iam::012345678912:role/test-role",
        "accountId": "012345678912",
        "userName": "test-role"
      },
      "webIdFederationData": {
        
      },
      "attributes": {
        "creationDate": "2022-12-08T08:15:10Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2022-12-08T08:54:43Z",
  "eventSource": "cloudtrail.amazonaws.com",
  "eventName": "LookupEvents",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "123.123.123.123",
  "userAgent": "aws-sdk-java/2.17.201 Linux/3.10.0-1160.80.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/17.0.2+8-LTS Java/17.0.2 vendor/Red_Hat__Inc. io/sync http/Apache cfg/retry-mode/legacy",
  "requestParameters": {
    "startTime": "Sep 9, 2022, 12:00:00 AM",
    "endTime": "Dec 8, 2022, 7:25:01 AM",
    "nextToken": "sNhgqKEs0ota607r7N/9sIrV2UdnOUs/1WWv/FTK1q/Mp6pFL4nm9olMGfiJOfh5t+9x7bxx23uh29du3hd93=="
  },
  "responseElements": null,
  "requestID": "5da63bf5-1234-1234-1234-b6b2bf2e114c",

continue in following comment.

Parth Dadhaniya

a year ago

  "eventID": "f72a6cf5-1234-1234-1234-1f5e135d0e88",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "012345678912",
  "eventCategory": "Management",
  "tlsDetails": {
    "tlsVersion": "TLSv1.2",
    "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "clientProvidedHostHeader": "cloudtrail.us-east-1.amazonaws.com"
  }
}

Parth Dadhaniya
a year ago
Hii there, any update from your side...?

Relevant content

Data loss or duplication on Cloudtrail LookupEvents API StartTime parameter
Accepted Answer
rePost-User-8310475
asked a year ago
Cloudtrail event notifications
Accepted Answer
Martin Halamicek
asked 2 years ago
Duplicate event fired
newza
asked a year ago
Can there be duplicate events if we increase Parallelization factor for processing DDB Stream events by Lambda ?
Arpan Pathak
asked 5 months ago
Why do I see duplicate messages in Amazon SQS for the same Amazon S3 event?
AWS OFFICIALUpdated 6 months ago
Why does the responseElements in some CloudTrail events for Secrets Manager contain "aRN" instead of "arn"?
AWS OFFICIALUpdated 9 months ago
What are the differences between data events and management events in CloudTrail?
AWS OFFICIALUpdated 2 years ago
How do I use CloudTrail LookupEvents to find the source of "Rate exceeded" errors associated with Lambda function management event API calls?
AWS OFFICIALUpdated a year ago
AWS B2B Data Interchange now publishes events to Amazon EventBridge
EXPERT
AWS
published a month ago
Free AWS Live Virtual Event: AWS Silicon Innovation Day
EXPERT
Markus Adhiwiyogo
published a year ago