How do I troubleshoot HTTP 403 Forbidden errors from an API Gateway custom domain name that requires mutual TLS?

7 minute read
0

My Amazon API Gateway custom domain name that has mutual Transport Layer Security (TLS) authentication activated returns HTTP 403 Forbidden errors. I don't know why this happens.

Short description

Note: API Gateway returns 403 Forbidden errors for a variety of reasons. This article addresses 403 Forbidden errors related only to mutual TLS. For information on troubleshooting other types of 403 Forbidden errors, see How do I troubleshoot HTTP 403 errors from API Gateway?

To invoke an API Gateway API with a custom domain name that requires mutual TLS, clients must present a trusted certificate in the API request. When a client invokes the API, API Gateway looks for the client certificate's issuer in your truststore.
The following conditions cause API Gateway to fail the TLS connection, and return a 403 status code:

  • API Gateway can't find the issuer of the client certificate in your truststore.
  • The client certificate uses an insecure signature algorithm.
  • The client certificate is self signed.

If Amazon CloudWatch logging is activated for your API, then an error message that indicates the cause of the error appears in your execution logs.

Important: If the API request doesn't produce any CloudWatch Logs after logging is activated, then the 403 Forbidden error isn't related to mutual TLS.

For REST APIs

If you set up Amazon CloudWatch logging for your REST API, then one of the following error messages also appears in your execution logs:

  • Access denied. Reason: Could not find issuer for certificate
  • Access denied. Reason: Client cert using an insecure Signature Algorithm
  • Access denied. Reason: self signed certificate

For HTTP API operations

HTTP APIs don't support execution logging. To troubleshoot 403 Forbidden errors returned by a custom domain name that requires mutual TLS and invokes an HTTP API, do the following:

  1. Create a new API mapping for your custom domain name that invokes a REST API for test purposes only.
    Note: If you don't have a REST API to test, then use the example PetStore REST API. Then, deploy the example API to a new stage, and create a new API mapping that uses your custom domain name.
  2. Follow the instructions in the Resolution section of this article with the new API mapping that you created to your REST API.
  3. Reroute the API mapping for your custom domain name back to your HTTP API.

Resolution

Confirm the cause of the error

  1. Turn on CloudWatch logging for your REST API.
  2. Configure execution and access logging.
    Note: When you configure access logging for this use case, it's a best practice to the following $context variables:
    { "accountId":"$context.accountId", "apiId":"$context.apiId", "domainName":"$context.domainName", "domainPrefix":"$context.domainPrefix", "error.message":"$context.error.message", "error.responseType":"$context.error.responseType", "extendedRequestId":"$context.extendedRequestId", "httpMethod":"$context.httpMethod", "identity.sourceIp":"$context.identity.sourceIp", "identity.clientCert.clientCertPem":"$context.identity.clientCert.clientCertPem", "identity.clientCert.subjectDN":"$context.identity.clientCert.subjectDN", "identity.clientCert.issuerDN":"$context.identity.clientCert.issuerDN", "identity.clientCert.serialNumber":"$context.identity.clientCert.serialNumber", "identity.clientCert.validity.notBefore":"$context.identity.clientCert.validity.notBefore", "identity.clientCert.validity.notAfter":"$context.identity.clientCert.validity.notAfter", "identity.userAgent":"$context.identity.userAgent", "path":"$context.path", "protocol":"$context.protocol", "requestId":"$context.requestId", "requestTime":"$context.requestTime", "requestTimeEpoch":"$context.requestTimeEpoch", "resourceId":"$context.resourceId", "resourcePath":"$context.resourcePath", "stage":"$context.stage", "responseLatency":"$context.responseLatency", "responseLength":"$context.responseLength", "status":"$context.status" }
    These variables tell API Gateway to generate CloudWatch Logs when your custom domain name that requires mutual TLS returns a 403 error. They also make it easier to identify the caller that tried to invoke your API operation.
  3. View your REST API's execution logs in CloudWatch to identify the cause of the error. If a 403 Forbidden error related to mutual TLS is logged, then you receive an error message similar to the following:
    Extended Request Id: {extendedRequestId} Access denied. Reason: {reason} 
    ForbiddenException Forbidden: {requestId}

Resolve "Access denied. Reason: Could not find issuer for certificate" errors

Verify that the issuer of the client certificate in the API request is included in the custom domain name's truststore

The issuer of the client certificate (client.pem) in the API request must be included in your custom domain name's truststore. The issuer must also be included as part of the certificate bundle (bundle.pem) in Amazon Simple Storage Service (Amazon S3).
To verify if the issuer of the client certificate is included in the required truststore, run the following OpenSSL command:

$ openssl verify -CAfile bundle.pem client.pem

-or-

If the certificate bundle contains intermediate certificate authorities, then run the following OpenSSL command:

$ openssl verify -CAfile rootCA.pem -untrusted intCA.pem client.pem

If the issuer of the client certificate in the API request is included in the required truststore, then the command returns an OK response.

If the issuer of the client certificate isn't included in the required truststore, then the command returns the following error: "error X at Y depth lookup: unable to get local issuer certificate"

Verify that all of the client certificates in your custom domain name's truststore are valid

If one of the client certificate's in your custom domain name's truststore isn't valid, then some clients might not be able to access your API. To confirm that the client certificates in your truststore are valid, do the following:

  1. Open the API Gateway console.

  2. In the left navigation pane, choose Custom domain names. Then, choose your custom domain name that requires mutual TLS.

  3. In the Details section, check for the following error message: There is an invalid certificate in your truststore bundle.

  4. If you see the error message, then decode the certificates in your truststore to identify which certificate produced the warning.
    Note: The following OpenSSL command displays the subject and contents of a certificate:

    $ openssl x509 -in certificate.crt -text -noout
  5. Update or remove the certificates that produced the warning. Then, upload a new truststore to Amazon S3.

For more information, see Troubleshooting certificate warnings.

Note: If their certificate chain is preserved, then API Gateway accepts client certificates signed directly by the root certificate authority or any other intermediate certificate authority. To validate client certificates signed by only the last intermediate certificate authority, use a request parameter-based AWS Lambda authorizer. You can use your custom validation algorithm at the Lambda function level. To do this, accept the client certificate as input from the API request.

Resolve "Access denied. Reason: Client cert using an insecure Signature Algorithm" errors

Verify that your truststore text file uses a supported hashing algorithm. API Gateway supports the following hashing algorithms in the truststore:

  • SHA-256 or stronger
  • RSA-2048 or stronger
  • ECDSA-256 or stronger

To confirm your truststore text file uses a supported hashing algorithm, run the following OpenSSL command:

$ openssl x509 -in client.crt -text -noout | grep 'Signature Algorithm'

The command response returns your truststore's signature algorithm.

For more information, see Configuring your truststore.

Resolve "Access denied. Reason: self signed certificate" errors

Verify that the self-signed client certificate in the API request isn't altered or corrupted. The client certification signing request (my_client.csr), client certificate private key (my_client.key), and the client certificate public key (my_client.pem) must match.

To compare moduli, run these OpenSSL commands:

$ openssl rsa -noout -modulus -in my_client.csr
$ openssl x509 -noout -modulus -in my_client.key
$ openssl x509 -noout -modulus -in my_client.pem

Note: To produce a shorter hash value for easier comparison, use a pipe on output modulus. See the following openssl sha1 example:

$ openssl [operation] -noout -modulus -in [data] | openssl sha1

A valid output example looks similar to the following:

2143831a73a8bb28467860df18550c696c03fbcb2143831a73a8bb28467860df18550c696c03fbcb
2143831a73a8bb28467860df18550c696c03fbcb

To confirm data integrity, verify that there isn't any data modification at the content level. Run the following diff command:

$ diff client.crt bundle.crt

For more information, see Configuring your truststore.

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago