Amazon VPC Lattice Troubleshooting Part 3 - Troubleshooting Target Connectivity
This is a series of 3 articles providing guidance in troubleshooting connectivity issues with Amazon VPC Lattice.
Note: This post is part 3 in a series and follows on from Part 1 - Establishing VPC Lattice Connectivity, which can be found here and Part 2 - Troubleshooting VPC Lattice Authentication, which can be found here
Table of Contents
- 3 Troubleshooting Target connectivity issues
3 Troubleshooting Target connectivity issues
The basic process to follow when troubleshooting target connectivity issues is:
- Isolate where the HTTP error code is being generated from, and diagnose that problem independently of VPC Lattice
- Use the VPC Lattice access logs failureReason field to identify the source of the error, either VPC Lattice or the target. failureReason will be set to a value other than - when VPC Lattice is generating the error.
- Use the returned headers to identify that the source of the failure is a host other than VPC Lattice. For example server: awselb/2.0 is a header generated from an ALB
- For non Lambda targets, Ensure your health checks are correct and showing healthy hosts.
- Check in all places upstream from VPC Lattice, including load balancers
- Verify the security group settings on each upstream hop
- Verify NACL settings on each upstream hop
- If the target is a load balancer, ensure the load balancer itself is reachable from VPC Lattice
- Verify the security group settings between VPC Lattice and the Load Balancer
- Verify NACL settings between VPC Lattice and the Load Balancer
3.1 EC2 Targets
EC2 Targets are reasonably straightforward to troubleshoot. We’ll use a similar process as we used in stage 1 to eliminate network connectivity issues, and then troubleshoot health check failures.
What good looks like
This is an example of a registered ec2 target, with a health check configured to return a 200 HTTP code.
In this case, I have the health check configured to /health.html which is a generic HTML page for the purposes of making sure our HTTP code is a 200.
Running curl from our client to test this via VPC Lattice shows the following:
sh-5.2$ curl -v https://webserver.example/health.html
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
* Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=webserver.example
* start date: Mar 14 23:17:50 2025 GMT
* expire date: Apr 14 00:17:50 2026 GMT
* subjectAltName: host "webserver.example" matched cert's "webserver.example"
* issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://webserver.example/health.html
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /health.html]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /health.html HTTP/2
> Host: webserver.example
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 200
< date: Fri, 21 Mar 2025 06:36:27 GMT
< server: Apache/2.4.62 (Amazon Linux)
< last-modified: Fri, 21 Mar 2025 06:29:06 GMT
< etag: "12-630d45f150830"
< accept-ranges: bytes
< content-length: 18
< content-type: text/html; charset=UTF-8
<
<body>
OK
</body>
* Connection #0 to host webserver.example left intact
sh-5.2$
3.1.1 EC2 Security Group or NACL Issue
Lets see what this looks like when we have a security group or NACL issue on the ec2 target host.
Health check is failing with a ConnectionTimeout. Connection Timeouts as we described above are usually related to network connectivity issues, so it could be security groups or NACLs. It could also be some fundamental issue with networking for that host, such as a broken Linux routing table, or iptables rules. It won’t be IP routing, because VPC Lattice automatically populates the routing table for subnets in the associated VPC.
With an assumption that it’s security groups or NACLs, you can verify that this is the case by looking again at our VPC Flow Logs.
Here’s an extract from a VPC flow log that shows this health check failure, the REJECT is a clear indication of a security group or NACL blocking the traffic.
2 012345678901 eni-0c39fa32e39819375 169.254.171.196 172.31.10.165 7873 80 6 2 120 1742552492 1742552559 REJECT OK
3.2 VPC Lattice to ALB to EC2 instance
In a case that you have an application load balancer behind VPC Lattice, which connects to a particular target, you now have two health checks that can be faulty. The first being the health check between VPC Lattice and the ALB (the VPC Lattice Target Group), and the second being the ALB to the ALB Target (the ALB Lattice Target Group).
You also have two sets of security groups to check - one being the ALB and the second being the EC2 target.
What good looks like
This is an example of a registered ALB target, pointing to an ec2 target. The ALB health check is configured to health.html on the ec2 host.
ALB Target Group:
VPC Lattice Target Group:
Note that the VPC Lattice Target Group does not have a health check when connected to an ALB, in the way that our EC2 target did.
3.2.1 VPC Lattice returning 504
I’ve deliberately blocked traffic from our ALB to our EC2 instance with a security group. Let’s see what is returned from VPC Lattice in this case:
sh-5.2$ curl https://webserver.example/health.html -v
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
* Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=webserver.example
* start date: Mar 14 23:17:50 2025 GMT
* expire date: Apr 14 00:17:50 2026 GMT
* subjectAltName: host "webserver.example" matched cert's "webserver.example"
* issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://webserver.example/health.html
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /health.html]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /health.html HTTP/2
> Host: webserver.example
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 504
< server: awselb/2.0
< date: Fri, 21 Mar 2025 11:03:12 GMT
< content-type: text/html
< content-length: 132
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
* Connection #0 to host webserver.example left intact
sh-5.2$
The following lines are the clue here:
< HTTP/2 504
< server: awselb/2.0
This indicates that VPC Lattice is receiving a 504 error from the ALB itself, because the server header is being generated from the ALB. This also demonstrates that VPC Lattice to ALB connectivity is functioning, which suggests we do not have a security group or NACL restriction on the ALB itself.
We can also validate this by attempting to connect to the ALB directly, from a host on the same subnet.
sh-5.2$ curl internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com -v
* Host internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com:80 was resolved.
* IPv6: (none)
* IPv4: 172.31.4.252, 172.31.29.182
* Trying 172.31.4.252:80...
* Connected to internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com (172.31.4.252) port 80
> GET / HTTP/1.1
> Host: internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
< Server: awselb/2.0
< Date: Fri, 21 Mar 2025 11:23:08 GMT
< Content-Type: text/html
< Content-Length: 132
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
* Connection #0 to host internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com left intact
In this case, an EC2 security group issue will cause the ALB to show our target as unhealthy.
In this scenario, you can also see the following in VPC Lattice access logs
{
"serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:servicenetwork/sn-0c460812811f4edc5",
"resolvedUser": "-",
"listenerType": "-",
"authDeniedReason": "-",
"targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:targetgroup/tg-078d2b5a0377e9a17",
"sourceVpcArn": "arn:aws:ec2:ap-southeast-2:012345678901:vpc/vpc-0957cde841630ab8f",
"destinationVpcId": "vpc-02085cbbe98e58637",
"sourceIpPort": "10.0.154.64:55072",
"listenerId": "-",
"targetIpPort": "172.31.4.252:80",
"failureReason": "-",
"serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:service/svc-00412bb2881b9a1b9",
"sourceVpcId": "vpc-0957cde841630ab8f",
"startTime": "2025-03-22T02:02:19Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 504,
"bytesReceived": 97,
"bytesSent": 759,
"duration": 10001,
"userAgent": "curl/8.5.0",
"hostHeader": "webserver.example",
"serverNameIndication": "webserver.example",
"requestToTargetDuration": 0,
"responseFromTargetDuration": 0,
"sslCipher": "TLS_AES_256_GCM_SHA384",
"tlsVersion": "TLSv1.3",
"callerPrincipal": "-",
"callerPrincipalOrgID": "-",
"callerX509IssuerOU": "-",
"callerX509SubjectCN": "-",
"callerX509SANNameCN": "-",
"callerX509SANDNS": "-",
"callerX509SANURI": "-"
}
Note there is no failureReason set, which indicates there is a problem after the ALB, and the 504 error is being returned from the ALB itself, not VPC Lattice.
3.2.1 VPC Lattice returning 503 - ALB to target issue
As with the above example, we need to look at the contents of the return code to determine what is causing the failure.
In this case, I’ve deliberately deregistered the target EC2 instance of the ALB. Let’s see what we get back from VPC Lattice:
sh-5.2$ curl -v https://webserver.example
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
* Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=webserver.example
* start date: Mar 14 23:17:50 2025 GMT
* expire date: Apr 14 00:17:50 2026 GMT
* subjectAltName: host "webserver.example" matched cert's "webserver.example"
* issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://webserver.example/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: webserver.example
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 503
< server: awselb/2.0
< date: Fri, 21 Mar 2025 11:30:01 GMT
< content-type: text/html
< content-length: 162
<
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>
* Connection #0 to host webserver.example left intact
We can see that the 503 message is coming back from server: awselb/2.0 indicating this is an ALB related error.
You can then verify from a system on the same subnet as the ALB that it is in fact the ALB generating this 503 error:
sh-5.2$ curl internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com -v
* Host internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com:80 was resolved.
* IPv6: (none)
* IPv4: 172.31.4.252, 172.31.29.182
* Trying 172.31.4.252:80...
* Connected to internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com (172.31.4.252) port 80
> GET / HTTP/1.1
> Host: internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 503 Service Temporarily Unavailable
< Server: awselb/2.0
< Date: Fri, 21 Mar 2025 11:32:24 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
<
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>
* Connection #0 to host internal-service-alb-1292922863.ap-southeast-2.elb.amazonaws.com left intact
In this scenario, you will also see that we do not have a failureReason set in the VPC Lattice access logs - which indicates that traffic from VPC Lattice to the ALB is successful.
{
"serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:servicenetwork/sn-0c460812811f4edc5",
"resolvedUser": "-",
"listenerType": "-",
"authDeniedReason": "-",
"targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:targetgroup/tg-078d2b5a0377e9a17",
"sourceVpcArn": "arn:aws:ec2:ap-southeast-2:012345678901:vpc/vpc-0957cde841630ab8f",
"destinationVpcId": "vpc-02085cbbe98e58637",
"sourceIpPort": "10.0.154.64:57326",
"listenerId": "-",
"targetIpPort": "172.31.4.252:80",
"failureReason": "-",
"serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:service/svc-00412bb2881b9a1b9",
"sourceVpcId": "vpc-0957cde841630ab8f",
"startTime": "2025-03-22T01:39:35Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 503,
"bytesReceived": 97,
"bytesSent": 789,
"duration": 2,
"userAgent": "curl/8.5.0",
"hostHeader": "webserver.example",
"serverNameIndication": "webserver.example",
"requestToTargetDuration": 1,
"responseFromTargetDuration": 0,
"sslCipher": "TLS_AES_256_GCM_SHA384",
"tlsVersion": "TLSv1.3",
"callerPrincipal": "-",
"callerPrincipalOrgID": "-",
"callerX509IssuerOU": "-",
"callerX509SubjectCN": "-",
"callerX509SANNameCN": "-",
"callerX509SANDNS": "-",
"callerX509SANURI": "-"
}
3.2.1 VPC Lattice returning 503 - VPC Lattice to ALB issue
In this scenario, we block traffic from VPC Lattice to the ALB with a NACL. You will also see a similar result with a security group block.
This is the flow log for the eni of the ALB
2 012345678901 eni-097f8ebc1750bbb70 169.254.171.195 172.31.4.252 9442 80 6 3 180 1742606800 1742606805 REJECT OK
And our curl result:
sh-5.2$ curl -v https://webserver.example
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
* Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=webserver.example
* start date: Mar 14 23:17:50 2025 GMT
* expire date: Apr 14 00:17:50 2026 GMT
* subjectAltName: host "webserver.example" matched cert's "webserver.example"
* issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://webserver.example/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: webserver.example
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 503
< content-length: 19
< content-type: text/plain
< date: Sat, 22 Mar 2025 01:26:46 GMT
<
* Connection #0 to host webserver.example left intact
The important thing to note here is that there is no Server: awselb/2.0 header, meaning the error message is coming from VPC Lattice itself.
In the VPC Lattice access logs, we see the following:
{
"serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:servicenetwork/sn-0c460812811f4edc5",
"resolvedUser": "-",
"listenerType": "-",
"authDeniedReason": "-",
"targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:targetgroup/tg-078d2b5a0377e9a17",
"sourceVpcArn": "arn:aws:ec2:ap-southeast-2:012345678901:vpc/vpc-0957cde841630ab8f",
"destinationVpcId": "vpc-02085cbbe98e58637",
"sourceIpPort": "10.0.154.64:32800",
"listenerId": "-",
"targetIpPort": "172.31.4.252:80",
"failureReason": "TargetConnectionError",
"serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:service/svc-00412bb2881b9a1b9",
"sourceVpcId": "vpc-0957cde841630ab8f",
"startTime": "2025-03-22T01:26:41Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 503,
"bytesReceived": 81,
"bytesSent": 646,
"duration": 5002,
"userAgent": "curl/8.5.0",
"hostHeader": "webserver.example",
"serverNameIndication": "webserver.example",
"requestToTargetDuration": 0,
"responseFromTargetDuration": 0,
"sslCipher": "TLS_AES_256_GCM_SHA384",
"tlsVersion": "TLSv1.3",
"callerPrincipal": "-",
"callerPrincipalOrgID": "-",
"callerX509IssuerOU": "-",
"callerX509SubjectCN": "-",
"callerX509SANNameCN": "-",
"callerX509SANDNS": "-",
"callerX509SANURI": "-"
}
The line "failureReason": "TargetConnectionError" is indicative of a connectivity error from VPC Lattice to the ALB itself, not the target of the ALB.
In comparison, when the ALB itself is not blocked, you will see an access log message like the following:
{
"serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:servicenetwork/sn-0c460812811f4edc5",
"resolvedUser": "-",
"listenerType": "-",
"authDeniedReason": "-",
"targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:targetgroup/tg-078d2b5a0377e9a17",
"sourceVpcArn": "arn:aws:ec2:ap-southeast-2:012345678901:vpc/vpc-0957cde841630ab8f",
"destinationVpcId": "vpc-02085cbbe98e58637",
"sourceIpPort": "10.0.154.64:57326",
"listenerId": "-",
"targetIpPort": "172.31.4.252:80",
"failureReason": "-",
"serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:service/svc-00412bb2881b9a1b9",
"sourceVpcId": "vpc-0957cde841630ab8f",
"startTime": "2025-03-22T01:39:35Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 503,
"bytesReceived": 97,
"bytesSent": 789,
"duration": 2,
"userAgent": "curl/8.5.0",
"hostHeader": "webserver.example",
"serverNameIndication": "webserver.example",
"requestToTargetDuration": 1,
"responseFromTargetDuration": 0,
"sslCipher": "TLS_AES_256_GCM_SHA384",
"tlsVersion": "TLSv1.3",
"callerPrincipal": "-",
"callerPrincipalOrgID": "-",
"callerX509IssuerOU": "-",
"callerX509SubjectCN": "-",
"callerX509SANNameCN": "-",
"callerX509SANDNS": "-",
"callerX509SANURI": "-"
}
Note that failureReason is not set in this case, indicating successful communication from VPC Lattice to the ALB.
3.3 VPC Lattice to ALB to Lambda Function
When connecting to a Lambda Function via ALB, the following size limits come into play: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html
3.3.1 VPC Lattice returning 413 from ALB
You may receive a 413 error code from the ALB when targeting a Lambda Function and exceeding the limits above. It is possible to verify that this is generated from the ALB using the same methodolody as above, by looking at the VPC Lattice access logs and inspecting the failureReason field.
In this scenario you can also use headers returned to the client, for example server: awselb/2.0 to determine this failure is generated from the upstream ALB.
A successful client connection from VPC Lattice to ALB to Lambda looks like the following:
sh-5.2$ curl https://webserver.example -v
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
* Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=webserver.example
* start date: Mar 14 23:17:50 2025 GMT
* expire date: Apr 14 00:17:50 2026 GMT
* subjectAltName: host "webserver.example" matched cert's "webserver.example"
* issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://webserver.example/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: webserver.example
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 200
< server: awselb/2.0
< date: Tue, 01 Apr 2025 22:58:06 GMT
< content-type: application/json
< content-length: 28
< set-cookie: cookies
<
* Connection #0 to host webserver.example left intact
This is an example of a failed connection, due to body size limits exceeding the maximum. In this example, I’ve sent a large json file, deliberately to exceed the ALB to Lambda size limit.
sh-5.2$ curl -H "Content-Type: application/json" -d @CloudFormationResourceSpecification2.json https://webserver.example -v
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
* Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=webserver.example
* start date: Mar 14 23:17:50 2025 GMT
* expire date: Apr 14 00:17:50 2026 GMT
* subjectAltName: host "webserver.example" matched cert's "webserver.example"
* issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
* SSL certificate verify ok.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://webserver.example/
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-type: application/json]
* [HTTP/2] [1] [content-length: 19738276]
> POST / HTTP/2
> Host: webserver.example
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 19738276
>
< HTTP/2 413
< server: awselb/2.0
< date: Tue, 01 Apr 2025 23:05:25 GMT
< content-type: text/html
< content-length: 148
* HTTP error before end of send, stop sending
<
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
</body>
</html>
* Connection #0 to host webserver.example left intact
sh-5.2$
When this fails, we will see VPC Lattice access logs indicating success, with "failureReason": "-" which is used to reflect that VPC Lattice could successfully connect to the target ALB.
{
"serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:servicenetwork/sn-0c460812811f4edc5",
"resolvedUser": "-",
"listenerType": "-",
"authDeniedReason": "-",
"targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:targetgroup/tg-078d2b5a0377e9a17",
"sourceVpcArn": "arn:aws:ec2:ap-southeast-2:012345678901:vpc/vpc-0957cde841630ab8f",
"destinationVpcId": "vpc-02085cbbe98e58637",
"sourceIpPort": "10.0.154.64:54808",
"listenerId": "-",
"targetIpPort": "172.31.22.32:80",
"failureReason": "-",
"serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:012345678901:service/svc-00412bb2881b9a1b9",
"sourceVpcId": "vpc-0957cde841630ab8f",
"startTime": "2025-04-01T23:05:25Z",
"requestMethod": "POST",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 413,
"bytesReceived": 426081,
"bytesSent": 826,
"duration": 4,
"userAgent": "curl/8.5.0",
"hostHeader": "webserver.example",
"serverNameIndication": "webserver.example",
"requestToTargetDuration": 0,
"responseFromTargetDuration": 0,
"sslCipher": "TLS_AES_256_GCM_SHA384",
"tlsVersion": "TLSv1.3",
"callerPrincipal": "-",
"callerPrincipalOrgID": "-",
"callerX509IssuerOU": "-",
"callerX509SubjectCN": "-",
"callerX509SANNameCN": "-",
"callerX509SANDNS": "-",
"callerX509SANURI": "-"
}
It is important to note that in this scenario, the VPC Lattice access logs report bytesReceived as less than the total size of the file. This is due to the ALB terminating the connection early, which means that VPC Lattice only sees part of the file transferred. For this reason, you should not use the bytesReceived measure as an indicator when connections are being terminated early, only when a connection has entirely completed.
Relevant content
- asked a year ago
- asked 4 months ago
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
