How do I troubleshoot issues with Network Firewall when a rule isn't working as expected?

Short description

The following scenarios can cause Network Firewall rules not to work as expected:

• Traffic isn't symmetrically routed through the firewall.
• The Network Firewall rule is incorrectly configured.

Note: Before you troubleshoot Network Firewall issues, confirm the following configurations:

• Firewall endpoints are deployed in their dedicated subnets and route tables. The firewall subnets must not have any workload resources.
• Route tables route traffic through the firewall endpoint. Check the Amazon CloudWatch metric ReceivedPackets to verify that the Network Firewall received packets. If the ReceivedPackets metric has a value of zero, then verify that the route tables have a correct route that points to the firewall endpoint.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, confirm that you're using the most recent AWS CLI version.

Verify symmetric routing through the firewall

For a centralized Network Firewall deployment model, turn on appliance mode for the transit gateway attachment of the inspection Virtual Private Cloud (VPC).

1. To check if appliance mode is turned on, run the following describe-transit-gateway-vpc-attachments AWS CLI command:

   aws ec2 describe-transit-gateway-vpc-attachments \
       --filters Name=transit-gateway-attachment-id,Values=tgw-attach-example
   

2. To turn on appliance mode, run the following modify-transit-gateway-vpc-attachment AWS CLI command:

   aws ec2 modify-transit-gateway-vpc-attachment \
       --transit-gateway-attachment-id tgw-attach-example \
       --options ApplianceModeSupport="enable"
   

   Note: Replace tgw-attach-example with the transit gateway attachment ID of the inspection VPC.

When you deploy a firewall endpoint in a public subnet, the internet gateway's ingress route table must route to the private subnet through the firewall endpoint. The private subnet's route table must also route internet traffic to the same firewall endpoint. For more information, see Simple single zone architecture with an internet gateway using AWS Network Firewall.

For a Multi-AZ firewall deployment, the internet gateway's ingress route table must route traffic to the private subnets through the same firewall endpoint. The route table must also be in the same Availability Zone as the private subnet. For more information, see Multi zone architecture with an internet gateway using AWS Network Firewall.

Fix an incorrectly configured Network Firewall rule

Issues with Network Firewall rules can occur because of how you configure the rule.

Stateless rules

For traffic to flow in both directions, apply stateless rules to both the ingress and egress rules. If you apply a rule on the ingress for incoming traffic, then you must also apply a rule to the egress for the response traffic. If you apply a rule to the egress for the response traffic, then you must also apply a rule on the ingress for incoming traffic. This rule configuration is similar to an access control list (ACL). For more information, see Working with stateless rule groups in AWS Network Firewall.

Stateless default actions for packets

Review the stateless default actions for both full packets and fragmented packets in your firewall policy. The default action settings determine how the firewall stateless engine handles individual packets when they don't match the stateless rules. Apply the stateless default action for fragmented packets to only IPv4 packet fragments that use the UDP transport protocol. For more information, see Firewall policy settings in AWS Network Firewall.

Note: The IPv6 protocol doesn't support fragmentation in the network. If a host sends a packet that's larger than the receiving host's MTU, then the receiving host drops the packet. Devices on the path also drop packets that are larger than the device's MTU.

Stateful rules

For the stateful rule engine to inspect traffic, the stateless rule engine must forward both traffic flow directions to stateful rule groups. For more information, see Network Firewall stateless and stateful rules engines.

Network Firewall rules must refer to private IP addresses

If a private subnet or workload has both public and private IP addresses, then the Network Firewall rule must refer to the private IP address.

Rule group variables

Stateful Network Firewall rule groups use the variable HOME_NET to define the scope of the firewall inspection. If you don't explicitly specify the HOME_NET variable, then the variable defaults to the VPC CIDR range where the Network Firewall is deployed. In this default setting, the Network Firewall inspects traffic that originates from sources within the VPC.

When you deploy a centralized Network Firewall, the HOME_NET variable of the rule group must include your remote VPC, or it must include the source CIDRs where your traffic originates. For more information, see Stateful domain list rule groups in AWS Network Firewall.

To inspect traffic from only certain private subnets and not the entire VPC, the HOME_NET variable must contain only the source subnet CIDRs. In the following example, the HOME_NET variable contains the source subnet CIDRs:

"RuleVariables": {
        "IPSets": {
           "HOME_NET": {
             "Definition": [
               "10.0.0.0/16",
               "10.1.0.0/16",
               "192.168.2.0/24"
             ]
           }
        }
    }

Use the DescribeRuleGroup API to check the stateful rule group details. If the RuleVariables object isn't in the response, then the firewall rule group uses default settings. The default settings for HOME_NET are set to the Network Firewall VPC CIDR.

Rule evaluation order

Applications that use TCP require the 3-way handshake to complete before the application sends requests. The first packet that the firewall sees in the connection is a TCP SYN from the source. If conflicting rules block the lower-level TCP packets, then the 3-way handshake doesn't succeed and the application times out.

For example, to detect the hostname, a domain list rule group uses the Host header in the HTTP request. It also uses the Server Name Indication (SNI) extension in the TLS handshake. For an HTTP request and TLS negotiation to happen, the initial 3-way TCP handshake on ports 80 (HTTP) and 443 (HTTPS) must first complete.

When you use rules that involve both application and transport layers, allow traffic on the TCP ports that the application requires. Review the rule evaluation order to verify that the preceding rules match the traffic.

With default action order, confirm that other rules aren't blocking traffic during evaluation. Use the flow keyword in your rules to avoid matching against lower-level protocol packets before higher-level application protocols are identified.

Review rule configuration examples

Incorrect rule configuration for TCP traffic

In the following example, the rule is incorrectly configured to block all outbound TCP connections and allow only HTTP requests to example.com. Because all the TCP traffic is dropped, the source client can't send the HTTP application request.

pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;)
drop tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Drop outbound TCP traffic"; sid:172192; rev:1;)

Correct configuration of TCP connection

In the following example, the "Allow TCP" rule establishes the TCP connection first. Then, the "Allowed HTTP domain" rule allows connections to the HTTP host example.com. The "Drop all other HTTP domains" rule blocks the rest of HTTP traffic, and the "Default Egress TCP Drop" rule blocks all other TCP traffic except for HTTP application layer traffic.

pass tcp $HOME_NET any -> $EXTERNAL_NET 80 (msg:"Allow TCP"; flow: not_established; sid:172190; rev:1;)
pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;)
drop http $HOME_NET any -> $EXTERNAL_NET 80 (msg:"Drop all other HTTP domains"; sid:172192; rev:1; flow: to_server;)
drop tcp $HOME_NET any -> any any (app-layer-protocol:!http; msg:"Default Egress TCP Drop"; flow:to_server; sid:172193;)

Note: Replace example.com with your target domain.

The Allow TCP (sid:172190) rule allows TCP connection establishment on port 80. It uses the flow: not_established keyword to permit the initial connection setup. The Allowed HTTP domain (sid:172191) rule permits HTTP traffic to example.com by inspecting the HTTP host header. The Drop all other HTTP domains (sid:172192) rule blocks HTTP traffic to domains other than example.com using the flow: to_server keyword. The Default Egress TCP Drop (sid:172193) rule blocks all TCP traffic that doesn't use HTTP as the application layer protocol using the app-layer-protocol:!http keyword.

Default action order automatically allows packets that don't match any defined rules.

With strict evaluation order, rule groups are evaluated in order of priority, starting with the lowest number. The rules in each rule group are processed in the order they're defined. Confirm that the configurations for the rules and manual priority order are correct. Don't match a lower priority rule against your network traffic. Write the rules based on the default action that you select on the strict firewall policy. For more information, see Managing evaluation order for Suricata compatible rules in AWS Network Firewall.

Incorrect configuration causing timeouts

In the following example, the specified configuration drops all TCP traffic by default before the firewall can detect the HTTP application-level request. This configuration causes unexpected timeouts.

Default Action: Drop all
Rule: pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;)

Correct configuration allowing TCP 3-way handshake

In the following example, the Network Firewall allows the TCP 3-way handshake to complete and allows only HTTP traffic for the domain example.com.

Default Action: Drop established
Rule: pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;)

For more firewall rule examples, see Examples of stateful rules for Network Firewall.

Related information

Deployment models for AWS Network Firewall

Defining rule actions in AWS Network Firewall

Domain list inspection for traffic from outside the deployment VPC