UDP client not receiving response from UDP server via AWS NAT Gateway

Question

Hello

I have a COAP client that sends a request to an AWS NLB, which then forwards the request to a COAP server by target group of type IP.
The server sends a response back to the client via a NAT Gateway, but the client is not receiving the response.

- I have confirmed from logs that the COAP server is receiving the request and sending the response back to the correct IP address and port number. I have also verified that the NAT Gateway is configured correctly and that it is forwarding the response back to the client.

- I have checked the security group settings for the NLB, UDP server, and NAT Gateway, and they all appear to be correct. I have also checked the network ACL settings and made sure that they are not blocking any traffic.

- I have checked the COAP client, and see that he receives response when the server is hosted in my laptop.

- I need to use the NAT gateway because I have to make some request from  server to the internet

**Details :**

**Assuming that the IP addresses are :**

Coap client : 
        public :  123.123.123.123

NLB : 
        private : 1.1.1.1

ec2 Instance were server is running : 
        private : 2.2.2.2

Nat gateway : 
        private : 3.3.3.3

---

**Logs in coap server when sending health request :**

CoapAssistanceResource - health request , exchange.getSourceAddress() is : /123.123.123.123
    CoapAssistanceResource - health request , exchange.getSourceSocketAddress() is : /123.123.123.123:40945
    CoapAssistanceResource - health request , exchange.getSourcePort() is : 40945
    CoapAssistanceResource - health request , exchange.getRequestCode() is : GET
    CoapAssistanceResource - health request , response is : 2.03

---

**Vpc flow logs with are :**

|${dstaddr} | ${dstport} | ${srcaddr} | ${srcport}  |${protocol} | ${action} | ${log-status} | ${type} | ${pkt-dstaddr}  |${pkt-srcaddr} | ${flow-direction}  |${traffic-path} |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| 2.2.2.2   |  5683    |   123.123.123.123   |   40945  |   17   |   ACCEPT   |   OK  |  IPv4  |   2.2.2.2                   |   123.123.123.123   |  ingress  |   -	| 
| 123.123.123.123   |  40945  |   2.2.2.2       |   5683    |  17    |   ACCEPT   |  OK   |  IPv4  |  123.123.123.123  |   2.2.2.2                     |   egress  |   1	| 
|  1.1.1.1   |   5683   |  123.123.123.123   |   40945   |  17    |   ACCEPT   |  OK   |  IPv4  |  1.1.1.1                     |  123.123.123.123  |   ingress  |   -	| 
| 123.123.123.123   |  40945  |  3.3.3.3       |   43099   |   17   |   ACCEPT   |  OK  |   IPv4  |   123.123.123.123  |   3.3.3.3       |   egress  |   8	|

I am not sure what else to check or how to troubleshoot this issue further. Any help or guidance would be greatly appreciated.

**Thank you!**

Accepted Answer

My guess is that there is some firewall (or other network security or NAT process) happening between the NLB and the client (perhaps a host firewall?) which is denying the inbound packet to the client (which is the reply packet from the server). Because that reply packet comes from a different IP address to the NLB (it comes from the NAT Gateway address) the client is unlikely to allow it. Again, this is a guess. You can probably confirm this by enabling VPC Flow Logs and seeing if the packet goes from the server to NAT Gateway and then to Internet Gateway.

I'm not sure why you are sending the reply packet directly to the client from the server though. Why not send the reply back via NLB? If you were using an instance target group then this would happen automatically. As you are using an IP target group you should send the packet back to the NLB IP address and it will then send it back to the client.

The server can still use the NAT Gateway to access other resources on the internet regardless of how it is configured with the NLB.

Destination	Target	Status	Propagated
0.0.0.0/0	nat-gateway-xxxxx	Active	NO
10.0.0.0/8	local	Active	NO
172.30.0.0/16	local	Active	NO

${dstaddr}	${dstport}	${srcaddr}	${srcport}	${protocol}	${action}	${log-status}	${type}	${pkt-dstaddr}	${pkt-srcaddr}	${flow-direction}	${traffic-path}
2.2.2.2	5683	123.123.123.123	40945	17	ACCEPT	OK	IPv4	2.2.2.2	123.123.123.123	ingress	-
123.123.123.123	40945	2.2.2.2	5683	17	ACCEPT	OK	IPv4	123.123.123.123	2.2.2.2	egress	1
1.1.1.1	5683	123.123.123.123	40945	17	ACCEPT	OK	IPv4	1.1.1.1	123.123.123.123	ingress	-
123.123.123.123	40945	3.3.3.3	43099	17	ACCEPT	OK	IPv4	123.123.123.123	3.3.3.3	egress	8

UDP client not receiving response from UDP server via AWS NAT Gateway

Relevant content