What are my options to analyze AWS WAF logs stored in CloudWatch or Amazon S3?

Resolution

To analyze and filter specific log requests, use Amazon CloudWatch Logs insights for CloudWatch logs or Amazon Athena for Amazon S3 logs.

Analyzing AWS WAF access logs with CloudWatch Logs insights

1. Open the Amazon CloudWatch console.
2. In the navigation pane, choose Logs, and then choose Log Insights.
3. For Select log group(s), choose one or more log groups to query that consist of AWS WAF access logs.
4. (Optional) Choose a time range for the period that you want to query.
5. Use query syntax to design queries.
6. Choose Run to view the results for the log group.

These are examples queries that you can use to filter out specific information for CloudWatch Logs Insights:

Filter on a specific string

Run this query to filter the log based on a specific string:
Note: Replace string {jndi:ldap. with the string you want to search.

fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.httpMethod as Method,httpRequest.uri as URI
| parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/
| parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/
| filter @message like "{jndi:ldap"
| sort action, URI desc

Filter by host

Run this query to filter by host:
Note: Replace Host value www.example.com with the Host you want to search.

parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/
| filter Host = "www.example.com"
| fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URI

Filter on POST requests

Run this query to isolate any POST requests:

parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/
| parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/
| fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.httpMethod as Method, httpRequest.uri as URI, httpRequest.requestId as RequestID
| filter httpRequest.httpMethod ="POST"
| display Rule, action, Country, ClientIP, Method, URI, Host, UserAgent, RequestID
| sort Rule, action desc

Filter on UserAgent

Run this query to filter by UserAgent:
Note: Replace User-Agent-Value with your UserAgent value**.**

parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/
| filter UserAgent like "<User-Agent-Value>"
| fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URI

Filter requests not originating from a country

Run this query to filter requests that don't originate from a specific country:

fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URI
| parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/
| parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/
| filter Country != "US"
| sort Country, action desc

Filter for cross-site scripting or SQL injection

Run this query to filter for cross-site scripting or SQL injection:

fields @timestamp, terminatingRuleId, action, httpRequest.clientIp as ClientIP, httpRequest.country as Country, terminatingRuleMatchDetails.0.conditionType as ConditionType, terminatingRuleMatchDetails.0.location as Location, terminatingRuleMatchDetails.0.matchedData.0 as MatchedData
| filter ConditionType in["XSS","SQL_INJECTION"]

Time series based on a terminating rule

Run this query to filter a time series based on a terminating rule:

#Time Series by Terminating Rule
filter terminatingRuleId = "AWS-AWSManagedRulesCommonRuleSet"
| stats count(*) as requestCount by bin(30m)

Summarize blocked requests by ClientIP, country, URI, and rule

Run this query to summarize blocked requests by ClientIP, country, URI, and rule:

fields httpRequest.clientIp as ClientIP, httpRequest.country as Country, httpRequest.uri as URI, terminatingRuleId as Rule
| filter action = "BLOCK"
| stats count(*) as RequestCount by Country, ClientIP, URI, Rule
| sort RequestCount desc

Top client IPs

Run this query to count the top client IPs:

stats count(*) as RequestCount by httpRequest.clientIp as ClientIP
| sort RequestCount desc

Top countries

Run this query to count the top countries:

stats count(*) as RequestCount by httpRequest.country as Country
| sort RequestCount desc

Top hosts

Run this query to count the top hosts:

parse @message /\{"name":"[Hh]ost\",\"value":\"(?<Host>[^"}]*)/
| stats count(*) as RequestCount by Host
| sort RequestCount desc

Top methods

Run this query to count the top methods:

stats count(*)as RequestCount by httpRequest.httpMethod as Method
| sort RequestCount desc

Top terminating rules

Run this query to count the top terminating rules:

stats count(*) as RequestCount by terminatingRuleId
| sort RequestCount desc

Top UserAgents

Run this query to count the top UserAgents:

parse @message /\{"name":"[Uu]ser\-[Aa]gent\",\"value\"\:\"(?<UserAgent>[^"}]*)/
| stats count(*) as RequestCount by UserAgent
| sort RequestCount desc

Request not terminated by Default_Action or rules with action ALLOW

Run this query to filter by requests not terminated by a Default_Action or rules with an ALLOW action:

fields @timestamp, terminatingRuleId, action, @message
| filter terminatingRuleId != 'Default_Action' and action != 'ALLOW'
| sort @timestamp desc

Request with invalid Captcha token

Run this query to filter by requests with an invalid Captcha token:

fields @timestamp, httpRequest.clientIp, httpRequest.requestId, captchaResponse.failureReason, @message
|filter captchaResponse.failureReason ='TOKEN_MISSING'
| sort @timestamp desc

Request block by rate-based rule

Run this query to filter by requests blocked by a rate-based rule:

fields @timestamp, httpRequest.clientIp, terminatingRuleId, httpRequest.country,@message
| filter terminatingRuleType ="RATE_BASED" ## and webaclId = "arn:aws:wafv2:us-east-1:xxxxxxxx:regional/webacl/waf-test/abcdefghijkl" ## uncomment to filter for specific WebACL
| sort requestCount desc

Filter all request detected by AWS Bot Control (ABC)

Run this query to filter all requests detected by ABC:

fields @timestamp, @message
|filter @message like 'awswaf:managed:aws:bot-control'
| parse @message '"labels":[*]' as Labels
| sort @timestamp desc

Analyzing AWS WAF access logs with Amazon Athena

You can turn on AWS WAF access logging directly to an Amazon S3 bucket. Or, you can use Amazon Kinesis Data Firehose delivery stream to deliver your AWS WAF access logs to an Amazon S3 bucket. To store logs in Amazon S3, see How do I configure AWS WAF comprehensive logging to store logs in Amazon S3?

When your access logs are in the Amazon S3 bucket, create the AWS WAF table to use Amazon Athena to query logs and filter various details.

These queries are examples that you can use to query AWS WAF logs with Athena:

Blocked requests with AWS WAF rule information

Run this Athena query to list all the blocked requests with AWS WAF rule:

SELECT timestamp,
    action,
    httpsourcename,
    httpsourceid,
    httprequest.requestID,
    httprequest.clientip,
    webaclid,
    terminatingruleid,
    terminatingruletype,
    rulegrouplist,
    terminatingrulematchdetails
FROM "wafv2"."waf_logs"
WHERE ("action" LIKE 'BLOCK')

Request User Agent

Run this Athena query to request the User Agent:
Note: Replace User-Agent with your UserAgent value.

select n.value, count(n.value) as count
from waf_logs
cross join
unnest(
  cast(
    httprequest.headers as ARRAY(ROW(name VARCHAR, value VARCHAR))
    )
  ) as x(n)
where n.name = 'User-Agent'
group by n.value
ORDER BY count(n.value) DESC

Request URI

Run this Athena query to check the request URI:

SELECT
"httprequest"."uri"
, "count"(*) "count"
FROM
  waf_logs
WHERE ("action" LIKE 'BLOCK')
GROUP BY "httprequest"."uri"
ORDER BY "count" DESC

Count blocked requests based on ClientIP

Run this Athena query to view the count of blocked requests based on ClientIP and country:

SELECT
  "httprequest"."clientip"
, "count"(*) "count"
, "httprequest"."country"
FROM
waf_logs
WHERE ("action" LIKE 'BLOCK')
GROUP BY "httprequest"."clientip", "httprequest"."country"
ORDER BY "count" DESC

View request count

Run this Athena query to view the request count:

SELECT 
  "httprequest"."clientip"
, "count"(*) "count"
,"httprequest"."country"
FROM
 waf_logs
WHERE ("action" LIKE
'BLOCK')
GROUP BY
"httprequest"."clientip", "httprequest"."country"
ORDER BY "count" DESC

For additional Athena query examples, see Example queries for AWS WAF logs.

---