How do I analyze AWS WAF logs in CloudWatch?

Resolution

To analyze and filter specific AWS WAF log requests in CloudWatch, use CloudWatch Logs Insights or the CloudWatch query generator.

Use CloudWatch Log Insights

You can use CloudWatch Log Insights in the CloudWatch console or in the AWS WAF console.

AWS WAF console

Complete the following steps:

1. Open the AWS WAF console.
2. In the navigation pane, choose Protection packs (web ACLs).
3. Choose your protection pack.
4. Choose View dashboard, logs and sampled requests.
5. In Log explorer, choose View in CloudWatch.
6. In the Query editor, enter your query. Use query syntax to design your queries. You can also choose queries from the Most frequently used queries list.
7. Choose Run query.

CloudWatch console

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, under Logs, choose Log Insights.
3. For Selection criteria, select one or more log groups to query. Or, choose Browse log groups, and then select your log groups.
4. (Optional) Choose a time range for your query.
5. Use query syntax to design your queries.
6. To view your results, choose Run query.

Use CloudWatch query generator

To use natural language to generate queries and analyze your access logs, run the query generator in CloudWatch.

Example queries 

To filter out specific information with CloudWatch Logs insights, use the following example queries.

Top client IP addresses

To count the top client IP addresses that access your application, run the following query:

stats count(*) as requestCount by httpRequest.clientIp
| sort requestCount desc

Top countries

To count top source countries that make requests to your application, run the following query:

stats count(*) as RequestCount by httpRequest.country as Country 
| sort RequestCount desc 

Top httpmethods

To count the top httpmethods that access your application, run the following query:

stats count(*)as RequestCount by httpRequest.httpMethod as Method
| sort RequestCount desc 

Top terminating rules

To count the top terminating rules in your logs, run the following query:

stats count(*) as RequestCount by terminatingRuleId
| sort RequestCount desc 

Top hosts

To count the top hosts that access your application, run the following query:

parse @message /\{"name":"[Hh]ost","value":"(?<Host>[^"]*)"/ 
| stats count(*) as RequestCount by Host 
| sort RequestCount desc 

Top user agents

To count the top user agents that access your application, run the following query:

parse @message /\{"name":"[Uu]ser\-[Aa]gent","value":"(?<UserAgent>[^"]*)"/  
| stats count(*) as RequestCount by UserAgent 
| sort RequestCount desc 

Top X-forwarded-Ips

To count the top XFF IP addresses that make requests to your application, run the following query:

parse @message /\{"name":"[Xx]-[Ff]orwarded-[Ff]or","value":"(?<XFF_IP>[^"}]*)/ 
| stats count(*) as RequestCount by XFF 
| sort RequestCount desc

Top IP addresses blocked by Ratebased Rule

To find IP addresses that breach the threshold and in the event of HTTP Flood DDoS, run the following query:

fields httpRequest.clientIp 
| filter terminatingRuleType = "RATE_BASED" ## and webaclId = "<Webacl ARN>" ## uncomment to filter for specific WebACL| 
| stats count(*) as requestCount by httpRequest.clientIp, httpRequest.country 
| sort requestCount desc 
| limit 100 

Filter by blocked requests

To filter for all blocked requests and their terminating rule, URI path, and client IP, run the following query:

fields @timestamp, httpRequest.clientIp as ClientIP, httpRequest.uri as URI, terminatingRuleId as rule 
| filter action = "BLOCK" 
| sort @timestamp desc 

Filter by host

To filter your logs by a specific host, run the following query:

fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URI 
| parse @message /\{"name":"[Hh]ost","value":"(?<Host>[^"]*)"/  
| filter Host = "www.example.com" 

Note: Replace www.example.com with the name of your host.

Filter by a specific string

To filter your logs by a specific string, run the following query:

fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.httpMethod as Method,httpRequest.uri as URI 
| parse @message /\{"name":"[Hh]ost","value":"(?<Host>[^"]*)"/ 
| parse @message /\{"name":"[Uu]ser\-[Aa]gent","value":"(?<UserAgent>[^"]*)"/  
| filter @message like "{jndi:ldap" 
| sort action, URI desc

Note: Replace {jndi:ldap with your string.

Filter by POST requests

To filter for POST requests, run the following query:

fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.httpMethod as Method, httpRequest.uri as URI 
| parse @message /\{"name":"[Uu]ser\-[Aa]gent","value":"(?<UserAgent>[^"]*)"/  
| parse @message /\{"name":"[Hh]ost","value":"(?<Host>[^"]*)"/  
| filter httpRequest.httpMethod ="POST" 
| display Rule, action, Country, ClientIP, Method, URI, Host, UserAgent 
| sort Rule, action desc 

NOTE: You can use the preceding query to filter by other HTTP methods. Replace POST with GET, Head, or other HTTP methods.

XSS or SQL Injection

To find patterns that cause cross-site scripting (XSS) or SQL Injection in the terminating rule for custom rules or AWS Managed Rule Groups, run the following query:

fields @timestamp  
| parse @message ',"terminatingRuleMatchDetails":[*],' as terminatingRuleMatchData  
| filter (terminatingRuleMatchData like /XSS/ or terminatingRuleMatchData like /SQL/)  
| display @timestamp, httpRequest.clientIp, httpRequest.country, terminatingRuleMatchData, httpRequest.requestId 

The query shows entries with a timestamp, client IP address, country of origin, match details, and the request Id.

Filter by country

To filter out requests that don't originate from a specific country, run the following query:

fields terminatingRuleId as Rule, action, httpRequest.country as Country, httpRequest.clientIp as ClientIP, httpRequest.uri as URI 
| parse @message /\{"name":"[Uu]ser\-[Aa]gent","value":"(?<UserAgent>[^"]*)"/  
| parse @message /\{"name":"[Hh]ost","value":"(?<Host>[^"]*)"/ 
| filter Country != "US" 
| sort Country, action desc 

Note: Replace US with the country code that you want to filter out.

Filter by requests counted by a specific rule in a rule group

To filter log entries for requests that a specific rule in a rule group counts and terminates by default, run the following query:

fields @timestamp 
| filter (@message like 'excludedRules":[{"exclusionType":"EXCLUDED_AS_COUNT","ruleId":"NoUserAgent_HEADER"}]}' and @message like 'terminatingRuleId":"Default_Action"') 
| parse @message '"ruleId":*}]}' as ruleMatchDetails 
| display @timestamp, httpRequest.clientIp, httpRequest.country, ruleMatchDetails, httpRequest.requestId 

Note: Replace ruleId with your rule Id.

Filter by requests with a CAPTCHA that's not valid

To filter for the top 100 requests with a CAPTCHA that's not valid, run the following query:

fields @timestamp, httpRequest.clientIp, httpRequest.requestId, captchaResponse.failureReason, @message 
| filter captchaResponse.failureReason ='TOKEN_MISSING' 
| sort @timestamp desc| limit 100 

Note: Replace 100 in the limit clause with the number of requests that you want to filter for.

This query shows the time of request, the IP address, the request Id, the response code, and the entire message.

Filter all request detected by ABC, ATP, ACFP, AntiDDOS AWS Managed Rules

To filter top 100 requests which matches ABC, ATP, ACFP or AntiDDOS AWS Managed Rules rule labels, run the following query:

fields @timestamp
| filter @message like 'awswaf:managed:aws:bot-control'
| display @timestamp,httpRequest.clientIp, httpRequest.uri,Labels, @message 
| sort @timestamp desc
| limit 100 

Note: Replace 100 in the limit clause with the number of requests that you want to filter for. Replace awswaf:managed:aws:bot-control with the rule group label. For ATP rule group, use awswaf:managed:aws:atp. For ACFP rule group, use awswaf:managed:aws:acfp. For AntiDDOS AMR rule group, use awswaf:managed:aws:anti-ddos.

To filter the logs with specific label include the following line in the preceding query:

| filter @message like 'awswaf:managed:aws:bot-control:signal:non_browser_user_agent'