Note: This post is part 2 in a series, Part 1 - Establishing VPC Lattice Connectivity, can be found here and Part 3 - Troubleshooting Target Connectivity can be found here

Table of Contents

• 2 Enabling VPC Lattice Access Logs
  • 2.1 Troubleshooting Auth Policy related issues
  • 2.2 Troubleshooting SigV4 signing issues
    • 2.2.1 Modification of signed headers after signing has occurred
    • 2.2.2 Credential must be scoped to a valid region
    • 2.2.3 URL Canonicalization
• Summary

2 Enabling VPC Lattice Access Logs

If we’re establishing a connection to VPC Lattice, but not seeing the target yet, we can look at the VPC Lattice access logs for more information. Access logs are also the way you are able to troubleshoot access policy related issues.

You can find details on enabling access logs here.

When you’ve enabled access logs to CloudWatch, you’ll be able to see log streams like the following:

Log streams can be enabled for the service network or the service itself. Looking at the service logs, we can see entries for connections to the target:

{
    "serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:1234567890:servicenetwork/sn-0c460812811f4edc5",
    "resolvedUser": "-",
    "listenerType": "-",
    "authDeniedReason": "-",
    "targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:1234567890:targetgroup/tg-096514b3475148e49",
    "sourceVpcArn": "arn:aws:ec2:ap-southeast-2:1234567890:vpc/vpc-0957cde841630ab8f",
    "destinationVpcId": "vpc-02085cbbe98e58637",
    "sourceIpPort": "10.0.154.64:32870",
    "listenerId": "-",
    "targetIpPort": "172.31.10.165:80",
    "failureReason": "-",
    "serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:1234567890:service/svc-00412bb2881b9a1b9",
    "sourceVpcId": "vpc-0957cde841630ab8f",
    "startTime": "2025-03-19T22:36:40Z",
    "requestMethod": "GET",
    "requestPath": "/",
    "protocol": "HTTP/2",
    "responseCode": 403,
    "bytesReceived": 211,
    "bytesSent": 672,
    "duration": 2,
    "userAgent": "curl/8.5.0",
    "hostHeader": "webserver.example",
    "serverNameIndication": "webserver.example",
    "requestToTargetDuration": 1,
    "responseFromTargetDuration": 0,
    "sslCipher": "TLS_AES_256_GCM_SHA384",
    "tlsVersion": "TLSv1.3",
    "callerPrincipal": "-",
    "callerPrincipalOrgID": "-",
    "callerX509IssuerOU": "-",
    "callerX509SubjectCN": "-",
    "callerX509SANNameCN": "-",
    "callerX509SANDNS": "-",
    "callerX509SANURI": "-"
}

2.1 Troubleshooting Auth Policy related issues

Here is an example of attempting to connect from the client to our VPC Lattice service, when the auth policy has been set using the template ‘Allow only authenticated access’

sh-5.2$ curl https://webserver.example  --insecure
AccessDeniedException: User: anonymous is not authorized to perform: vpc-lattice-svcs:Invoke on resource: arn:aws:vpc-lattice:ap-southeast-2:1234567890:service/svc-00412bb2881b9a1b9/ because no service-based policy allows the vpc-lattice-svcs:Invoke actionsh-5.2$
sh-5.2$

The access log also shows the details for this connection:

{
    "serviceNetworkArn": "arn:aws:vpc-lattice:ap-southeast-2:1234567890:servicenetwork/sn-0c460812811f4edc5",
    "resolvedUser": "Anonymous",
    "listenerType": "-",
    "authDeniedReason": "Service",
    "targetGroupArn": "arn:aws:vpc-lattice:ap-southeast-2:1234567890:targetgroup/tg-096514b3475148e49",
    "sourceVpcArn": "arn:aws:ec2:ap-southeast-2:1234567890:vpc/vpc-0957cde841630ab8f",
    "destinationVpcId": "vpc-02085cbbe98e58637",
    "sourceIpPort": "10.0.154.64:45642",
    "listenerId": "-",
    "targetIpPort": "-",
    "failureReason": "ClientAccessDenied",
    "serviceArn": "arn:aws:vpc-lattice:ap-southeast-2:1234567890:service/svc-00412bb2881b9a1b9",
    "sourceVpcId": "vpc-0957cde841630ab8f",
    "startTime": "2025-03-19T22:42:27Z",
    "requestMethod": "GET",
    "requestPath": "/",
    "protocol": "HTTP/2",
    "responseCode": 403,
    "bytesReceived": 82,
    "bytesSent": 390,
    "duration": 16,
    "userAgent": "curl/8.5.0",
    "hostHeader": "webserver.example",
    "serverNameIndication": "webserver.example",
    "requestToTargetDuration": 0,
    "responseFromTargetDuration": 0,
    "sslCipher": "TLS_AES_256_GCM_SHA384",
    "tlsVersion": "TLSv1.3",
    "callerPrincipal": "Anonymous",
    "callerPrincipalOrgID": "-",
    "callerX509IssuerOU": "-",
    "callerX509SubjectCN": "-",
    "callerX509SANNameCN": "-",
    "callerX509SANDNS": "-",
    "callerX509SANURI": "-"
}

This is failing, because curl by default does not provide any authentication credentials, which should be provided by the SigV4 or SigV4A signing process.

You can test SigV4 authentication with SigV4 signing using curl, as follows:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
CREDENTIALS=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/ssmprofile)
AWS_ACCESS_KEY_ID=$(echo $CREDENTIALS| jq -r .AccessKeyId)
AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIALS | jq -r .SecretAccessKey)
AWS_SESSION_TOKEN=$(echo $CREDENTIALS | jq -r .Token)
AWS_REGION=ap-southeast-2

curl -v https://webserver.example/ --insecure \
  --aws-sigv4 aws:amz:${AWS_REGION}:vpc-lattice-svcs \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
  --header 'Accept: application/json' \
  --header "x-amz-content-sha256: UNSIGNED-PAYLOAD" 

This command retrieves credentials from the instance metadata service on an EC2 host, then uses curl and SigV4 to sign the request and connect to the service.

For connectivity to VPC Lattice services we need to set the x-amz-content-sha256 header to the string UNSIGNED-PAYLOAD as per https://docs.aws.amazon.com/vpc-lattice/latest/ug/sigv4-authenticated-requests.html due to VPC Lattice not supporting payload signing.

Sample output from running this command is below:

curl -v https://webserver.example/ --insecure \
  --aws-sigv4 aws:amz:${AWS_REGION}:vpc-lattice-svcs \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
  --header 'Accept: application/json' \
  --header "x-amz-content-sha256: UNSIGNED-PAYLOAD"
* Host webserver.example:443 was resolved.
* IPv6: fd00:ec2:80::a9fe:ab01
* IPv4: 169.254.171.1
*   Trying 169.254.171.1:443...
* Connected to webserver.example (169.254.171.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=webserver.example
*  start date: Mar 14 23:17:50 2025 GMT
*  expire date: Apr 14 00:17:50 2026 GMT
*  issuer: C=AU; O=LatticeSolution; OU=LatticeSolution; dnQualifier=LatticeSolution; ST=LatticeSolution; CN=LatticeSolution; serialNumber=12345
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* Server auth using AWS_SIGV4 with user 'ASIAEXAMPLE'
* [HTTP/2] [1] OPENED stream for https://webserver.example/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: webserver.example]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [authorization: AWS4-HMAC-SHA256 Credential=ASIAEXAMPLE/20250401/ap-southeast-2/vpc-lattice-svcs/aws4_request, SignedHeaders=accept;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=a1234567]
* [HTTP/2] [1] [x-amz-date: 20250401T232313Z]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [x-amz-security-token: a1234567]
* [HTTP/2] [1] [accept: application/json]
* [HTTP/2] [1] [x-amz-content-sha256: UNSIGNED-PAYLOAD]
> GET / HTTP/2
> Host: webserver.example
> Authorization: AWS4-HMAC-SHA256 Credential=ASIAEXAMPLE/20250401/ap-southeast-2/vpc-lattice-svcs/aws4_request, SignedHeaders=accept;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=a1234567
> X-Amz-Date: 20250401T232313Z
> User-Agent: curl/8.5.0
> x-amz-security-token: a1234567
> Accept: application/json
> x-amz-content-sha256: UNSIGNED-PAYLOAD
>
< HTTP/2 200
< server: awselb/2.0
< date: Tue, 01 Apr 2025 23:23:13 GMT
< content-type: application/json
< content-length: 28
< set-cookie: cookies
<

This gives the output:

AccessDeniedException: User: arn:aws:sts::1234567890:assumed-role/ssmprofile/i-0ed9ed6c3f6666a0b is not authorized to perform: vpc-lattice-svcs:Invoke on resource: arn:aws:vpc-lattice:ap-southeast-2:1234567890:service/svc-00412bb2881b9a1b9/ because no identity-based policy allows the vpc-lattice-svcs:Invoke action

When we receive a message like this back from VPC Lattice, it indicates that the SigV4 signing process has been successful and VPC Lattice is able to identify the caller. In this case, the caller has been identified as arn:aws:sts::1234567890:assumed-role/ssmprofile/i-0ed9ed6c3f6666a0b which is the instance profile we’ve assigned to the client.

Based on these, you’re able to troubleshoot where the policy violation may be occurring, it can be on the role that is making the connection (the instance profile), the service network auth policy, or the service auth policy.

Check each of these policies one by one, and ensure that you've permitted the vpc-lattice-svcs:Invoke action, and you've specified the resource correctly in the policy statement. To identify further where it is failing, you can look in the VPC Lattice access logs in the authDeniedReason to determine whether a service network or service policy may be causing the failure.

2.2 Troubleshooting SigV4 signing issues

You may come across issues relating to your signing code when enabling authentication on a VPC Lattice service.

A SigV4/SigV4A signing issue is identified by a message such as the following:

InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

To troubleshoot signing issues, it is useful to use curl with SigV4 signing as you saw in section 2.1 above.

2.2.1 Modification of signed headers after signing has occurred

The SigV4 signing process creates a list of HTTP headers included in the signature and places this list in the Authorization header, in the SignedHeaders field. If any of the HTTP headers listed in this field are modified after signing has occurred, you will receive an InvalidSignature message. This is by design - one purpose of SigV4 signing is to assert that the content sent to a SigV4 enabled service has not been modified.

A common reason for this is when your signing code is independent from code used to send the request to VPC Lattice, and the sending code modifies one or more of the SignedHeaders as it sends the request. Some reasons this might occur are:

• A proxy between the source and destination modifies a header such as Host
• The sending code modifies the Host or other header during sending

To troubleshoot these scenarios, use the curl example above to generate a successful connection. Then compare the headers being sent in the curl example, to the headers being used by and generated by your signing code. It is useful to print out the HTTP payload before you perform signing, then as you send the request to the wire. In particular look for the following:

• You have a valid AWS Access Key ID (the first segment of the Credential= section of the Authorization header, before the first /
• X-Amz-Date is correct
• X-Amz-Security-Token is valid
• Any headers listed in SignedHeaders are the same before signing and as you send the request to the wire.
• Authorization contains a calculated Signature field

2.2.2 Credential must be scoped to a valid region

If you see a message such as:

InvalidSignatureException: Credential should be scoped to a valid region.

it can be an indication that you are using an invalid region string in your SigV4 signature calculation. The region used in SigV4 calculation should be the region of the VPC Lattice service. In the curl example above, we specify the region in the following curl parameter:

--aws-sigv4 aws:amz:${AWS_REGION}:vpc-lattice-svcs

2.2.3 URL Canonicalization

If you are creating your own signing code, you may come across issues related to the canonicalization of URLs. For example, you'll have signing issues if you don't perform URL Encoding in accordance with the SigV4 specification, or you incorrectly convert characters such as spaces to + (a plus sign). The safest way to ensure you are signing correctly is to use the AWS SDK, or the AWS C Auth Runtime (https://github.com/awslabs/aws-c-auth), which has specific processing to handle URL encoding. There are samples available for many languages here. However, if you need to write your own code, you need to follow the steps listed here exactly.

Summary

In this article we've covered how to troubleshoot Amazon VPC Lattice related connectivity issues when you are able to reach the service from your client. In Part 3 - Troubleshooting Target Connectivity, we'll explain how to troubleshoot upstream connection issues, such as issues with target groups and upstream load balancing.