How can I retrieve and then analyze my CloudTrail Logs with CloudWatch Logs Insights?

9 minute read
0

I want to use Amazon CloudWatch Logs Insights to retrieve and then analyze my Amazon CloudTrail Logs.

Short description

After you configure CloudTrail to log CloudWatch Logs, you can use queries in CloudWatch Logs Insights to retrieve the CloudTrail logs. Then, you can monitor specific account activity.

Resolution

Use the following queries to retrieve CloudWatch Logs to analyze and explore Amazon Simple Storage Service (Amazon S3) bucket and object activity. Note that by default, CloudTrail doesn't capture Amazon S3 data events. You must turn on event logging in CloudTrail to retrieve event logs for S3 buckets and objects.

You can build on these example queries to create additional and more complex Logs Insights queries that align to your use case. You can also integrate queries with your CloudWatch dashboard to visualize your queries as charts and graphs alongside related metrics.

Query 1: Latest events

Objective

Retrieve the most recent CloudTrail Log events with the default @timestamp and @message fields.

Query

#Retrieve the most recent CloudTrail events
fields @timestamp, @message
| sort @timestamp desc
| limit 2

Results

@timestamp@message
2022-02-18 17:52:31.118{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","principalId":"AROAWZKRRJU47ARZN7ECC:620d7d78144334d6933c27195cae2a98", "arn":"arn:aws:sts::123456789012:assumed- role/Amazon_EventBridge_Invoke_Run_Command_371790151/620d7d78144334d6933c27195cae2a98","accountId":"123456789012", "accessKeyId":"ASIAWZKRRJU4Y45M4SC6","sessionContext":{"sessionIssuer": {"type":"Role","principalId":"AROAWZKRRJU47ARZN7ECC","arn":"arn:aws:iam::123456789012:role/service- role/Amazon_EventBridge_Invoke_Run_Command_371790151","accountId":"123456789012","userName": "Amazon_EventBridge_Invoke_Run_Command_371790151" (output truncated)
2022-02-18 17:51:52.137{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","principalId":"AROAWZKRRJU43YP4FHR2N:StateManagerService","arn":"arn:aws:sts::123456789012:assumed-role/AWSServiceRoleForAmazonSSM/StateManagerService","accountId":"123456789012","sessionContext":{"sessionIssuer":{"type":"Role","principalId":"AROAWZKRRJU43YP4FHR2N","arn":"arn:aws:iam::123456789012:role/aws-service-role/ssm.amazonaws.com/AWSServiceRoleForAmazonSSM","accountId":"123456789012","userName":"AWSServiceRoleForAmazonSSM"}, "webIdFederationData":{},"attributes":{"creationDate":"2022-02-18T17:50:06Z","mfaAuthenticated":"false"}},"invokedBy":"ssm.amazonaws.com"},"eventTime":"2022-02-18T17:50:06Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"eu-west-1","sourceIPAddress":"ssm.amazonaws.com","userAgent":"ssm.amazonaws.com","requestParameters":{"maxResults":50,"instancesSet": (output truncated)

Query 2: Break out individual fields

Objective

  • Isolate the individual fields in @message.
  • Display selected fields in the CloudTrail event.

Query

#Breakout Individual Fields
fields @timestamp, awsRegion, eventCategory, eventSource, eventName, eventType, sourceIPAddress, userIdentity.type
| sort @timestamp desc
| limit 2

Results

@timestampawsRegioneventCategoryeventSourceeventNameeventTypesourceIPAddressuserIdentity.type
2022-02-18 18:00:09.647ca-central-1Managementsts.amazonaws.comAssumeRoleAwsApiCallcloudtrail.amazonaws.comAWSService
2022-02-18 18:00:09.647ca-central-1Managementsts.amazonaws.comAssumeRoleAwsApiCallcloudtrail.amazonaws.comAWSService

Query 3: Filter by Amazon Elastic Compute Cloud (Amazon EC2) run instances

Objective

  • Retrieve specific fields in the CloudTrail event.
  • Rename fields to use more meaningful labels.
  • Filter the latest EC2 instances launched in this account based on the API call.

Query

#EC2: Recently Launched Instances
fields eventTime, eventName as API, responseElements.instancesSet.items.0.instanceId as InstanceID, userIdentity.sessionContext.sessionIssuer.type as IssuerType, userIdentity.type as IdentityType, userIdentity.sessionContext.sessionIssuer.userName as userName
| filter eventName = 'RunInstances'
| sort eventTime desc
| limit 2

Results

eventTimeAPIInstanceIDIssuerTypeIdentityTypeuserName
2022-02-18T17:36:38ZRunInstancesi-0325b4d6ae4e93c75RoleAssumedRoleAWSServiceRoleForAutoScaling
2022-02-18T13:45:18ZRunInstancesi-04d17a8425b7cb59aRoleAssumedRoleAWSServiceRoleForAutoScaling

Query 4: Filter by the most recent console login

Objective

  • Retrieve specific fields in the CloudTrail event.
  • Rename fields to use more meaningful labels.
  • Filter the latest logins in the console based on the API call.

Query

#Console Login: Most Recent API Calls
fields eventTime, eventName, responseElements.ConsoleLogin as Response, userIdentity.arn as ARN, userIdentity.type as User_Type
| filter eventName = 'ConsoleLogin'
| sort eventTime desc
| limit 10

Results

eventTimeeventNameResponseARNUser_Type
2022-02-18T17:35:44ZConsoleLoginSuccessarn:aws:iam::123456789012:user/test_userIAMUser
2022-02-17T13:53:58ZConsoleLoginSuccessarn:aws:sts::123456789012:assumed-role/Admin/test_userAssumedRole

Query 5: Filter by the console login with failed authentication

Objective

  • Retrieve specific fields in the CloudTrail event.
  • Rename fields to use more meaningful labels.
  • Filter the latest unsuccessful logins from the console.

Query

#ConsoleLogin: Filter on Failed Logins
fields eventTime, eventName, responseElements.ConsoleLogin as Response, userIdentity.userName as User, userIdentity.type as User_Type, sourceIPAddress, errorMessage
| filter eventName = 'ConsoleLogin' and responseElements.ConsoleLogin = 'Failure'
| sort eventTime desc
| limit 10

Results

eventTimeeventNameResponseUserUser_TypesourceIPAddresserrorMessage
2022-02-18T20:10:55ZConsoleLoginFailureechoIAMUser12.34.56.89Failed authentication
2022-02-18T20:10:43ZConsoleLoginFailureechoIAMUser12.34.56.89Failed authentication

Query 6: Filter by Amazon Simple Storage Service (Amazon S3) object upload

Objective

  • Retrieve specific fields in the CloudTrail event.
  • Rename fields to use more meaningful labels.
  • Filter on the API call and the target S3 bucket.

Query

#Filter PutObject API Calls on a specific S3 Bucket
fields @timestamp, eventName as API, requestParameters.bucketName as BucketName, requestParameters.key as Key, userIdentity.sessionContext.sessionIssuer.userName as UserName
| filter eventName = 'PutObject' and BucketName = 'target-s3-bucket'
| sort @timestamp desc
| limit 2

Results

@timestampAPIBucketNameKeyUserName
2022-02-12 17:16:07.415PutObjecttest_bucket1w4r9Hg4V7g.jpg
2022-02-12 16:29:43.470PutObjecttest_bucket26wyBy0hBoB.jpg

Query 7: Summarize S3 activity

Objective

  • Filter based on the Amazon S3 service.
  • Aggregate all the matching events based on the count statistic.
  • Divide the results based on the API, S3 bucket, and key.
  • Use the stats command to rename the fields.
  • Sort by descending order.

Query

#S3 Activity: Bucket Key Details
filter eventSource = 's3.amazonaws.com'
| stats count(*) as Hits by eventName as API, requestParameters.bucketName as BucketName, requestParameters.key as Key
| sort Hits desc
| limit 5

Results

APIBucketNameKeyHits
ListAccessPoints44
GetBucketAclteam1-ctrail-multi-region27
GetBucketAclteam2-dub-cloudtrail27
GetBucketAclaws-cloudtrail-logs-123456789012-ba940dd726
GetObjectdevsupport-prodrdscr/individual/12345678901218

Query 8: Summarize AWS KMS decrypt activity

Objective

  • Filter based on the AWS Key Management Service (AWS KMS) service and Decrypt API.
  • Use the fields command to rename fields, and then aggregate on the user-friendly names.
  • Aggregate all the matching events based on the count statistic.
  • Divide the results based on the AWS KMS key and user.
  • Sort by descending order.

Query

#KMS Decrypt Activity: Key User Details
fields resources.0.ARN as KMS_Key, userIdentity.sessionContext.sessionIssuer.userName as User
| filter eventSource='kms.amazonaws.com' and eventName='Decrypt'
| stats count(*) as Hits by KMS_Key, User
| sort Hits desc
| limit 2

Results

KMS_KeyUserHits
arn:aws:kms:us-east-1:123456789012:key/03f2923d-e213-439d-92cf-cbb444bd85bdAWSServiceRoleForConfig12
arn:aws:kms:us-east-1:123456789012:key/03f2923d-e213-439d-92cf-cbb444bd85bdFoxTrot-1UQJBODTWZYZ68

Query 9: Summarize API calls with errors

Objective

  • Filter based on the presence of the errorCode field.
  • Aggregate all the matching events based on the count statistic.
  • Divide the results based on the AWS Service, API, and errorCode.
  • Use the stats command to rename the fields.
  • Sort by largest number of matches.

Query

#Summarize API Calls with Errors
filter ispresent(errorCode)
| stats count(*) as Num_of_Events by eventSource as AWS_Service, eventName as API, errorCode
| sort Num_of_Events desc
| limit 5

Results

AWS_ServiceAPIerrorCodeNum_of_Events
s3.amazonaws.comGetBucketPublicAccessBlockNoSuchPublicAccessBlockConfiguration79
lambda.amazonaws.comGetLayerVersionPolicy20181031ResourceNotFoundException66
s3.amazonaws.comGetBucketPolicyStatusNoSuchBucketPolicy60
s3.amazonaws.comHeadBucketAccessDenied47
logs.amazonaws.comCreateLogStreamResourceNotFoundException21

Query 10: Summarize S3 API calls with error codes

Objective

  • Filter based on the Amazon S3 service and the presence of errorCode field.
  • Aggregate all the matching events based on the count statistic.
  • Divide the results based on errorCode and errorMessage.
  • Sort by largest number of matches.

Query

#S3: Summarize Error Codes
filter eventSource = 's3.amazonaws.com' and ispresent(errorCode)
| stats count(*) as Hits by errorCode, errorMessage
| sort Hits desc
| limit 5

Results

errorCodeerrorMessageHits
AccessDeniedAccess Denied86
NoSuchBucketPolicyThe bucket policy does not exist80
NoSuchPublicAccessBlockConfigurationThe public access block configuration was not found79
ObjectLockConfigurationNotFoundErrorObject Lock configuration does not exist for this bucket3
ServerSideEncryptionConfigurationNotFoundErrorThe server side encryption configuration was not found3

Query 11: Summarize AccessDenied/UnauthorizedOperation API calls by AWS service, API, and AWS Identity and Access Management (IAM) user

Objective

  • Filter on either AccessDenied or UnauthorizedOperation CloudTrail events.
  • Aggregate all the matching events based on count statistic.
  • Divide the results based on the errorCode, AWS Service, API, and IAM user or role.
  • Use the stats command to rename fields.
  • Sort in descending order.

Query

#Summarize AccessDenied/UnauthorizedOperation API Calls by AWS Service, API, IAM User
filter (errorCode='AccessDenied' or errorCode='UnauthorizedOperation')
| stats count(*) as NumberOfEvents by errorCode, eventSource as AWS_Service, eventName as API, userIdentity.type as IdentityType, userIdentity.invokedBy as InvokedBy
| sort NumberOfEvents desc
| limit 10

Results

errorCodeAWS_ServiceAPIIdentityTypeInvokedByNumberOfEvents
AccessDenieds3.amazonaws.comHeadBucketAWSServicedelivery.logs.amazonaws.com83
AccessDenieds3.amazonaws.comGetObjectAssumedRole9

Query 12: AWS KMS hourly call volume

Objective

  • Filter based on the AWS KMS service and Decrypt API.
  • Aggregate all the matching events into one-hour bins.
  • Visualize the results on a line graph.

Query

#KMS: Hourly Decrypt Call Volume
filter eventSource='kms.amazonaws.com' and eventName='Decrypt'
| stats count(*) as Hits by bin(1h)

Results

bin(1h)Hits
2022-02-18 19:00:00.00016
2022-02-18 18:00:00.00025
2022-02-18 17:00:00.00028
2022-02-18 16:00:00.00014
2022-02-18 15:00:00.00016

Related information

Monitor AWS CloudTrail log data in Amazon CloudWatch (Video)

Add a query to dashboard or export query results