By using AWS re:Post, you agree to the AWS re:Post Terms of Use

Inconsistent api gateway SSL answer

0

Context : I have the following "simple" infrastructure : client --- mTLS request --> api gateway --> lambda

Test #1 : mTLS request: I've tested a loop of 50 calls on my API with mTLS, and depending of the environment I have between 0% of errors to 20% errors, with the following error return :

for i in range(1,50):
    resp = requests.post('https://subdomain.domain/dp',headers=headers, data=None,cert=CERT_TUPLE)

with the following exception :

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The following repost topic suggest an ssl error : https://repost.aws/questions/QUhIDK9NkpSJiF4l-jmqgfmQ/api-gateway-intermittent-ssl-connect-connection-reset-by-peer I have investigated this way with the following test :

Test #2 :openssl test:

for i in {1..50}
do
openssl s_client -connect subdomain.domain:443 -servername domain>/dev/null 2>&1
status=$?
echo "status code : $status"
done

With this command I have also **multiple ** returned code : 1 and 104 and the negotiation is different between a correct negociation and an incorect negociation.

Correct negociation :

Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA224:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ECDSA+SHA224:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA224:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ECDSA+SHA224
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 4325 bytes and written 454 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: B81448EC098C8B296B652D984E4188EC3A4381626F97656502A78B78D3FDC83B
    Session-ID-ctx:
    Master-Key: 443723FE82C74D633F6809115839B660A78B36CBE2FAD0069DFB042AFA0A4AA77946FFC9279C01A5EA4515A1D95FFAF1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1732689901
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

Incorrect negociation:

Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA224:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ECDSA+SHA224:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4442 bytes and written 410 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

In the both examples, the call is identical and done one after the other.

  • Could it be an inconcistancy in the aws api gateway loadbalancing farm ?
  • Have you already seen this error ?
  • How can I bypass this error ?

Regards,

1 Answer
-1

The inconsistent behavior you're experiencing with API Gateway SSL connections is not uncommon and can be attributed to several factors. Let's address your questions and provide some insights:

  1. Inconsistency in AWS API Gateway load balancing: While it's possible that load balancing could contribute to the inconsistency, it's unlikely to be the primary cause of the SSL negotiation issues you're seeing. API Gateway is designed to handle SSL/TLS connections consistently across its infrastructure.

  2. Familiarity with this error: The error you're encountering is indeed a known issue that can occur with API Gateway, particularly when dealing with SSL/TLS connections and mTLS (mutual TLS) setups.

  3. Bypassing the error:

To address and potentially bypass this error, consider the following approaches:

a) Certificate chain verification: Ensure that the backend (Lambda in your case) is providing a complete certificate chain, including intermediate certificates. API Gateway expects the certificate chain to be intact, containing the root CA, intermediate CAs, and the parent certificate details.

b) Certificate validity: Verify that all certificates in use are valid and not expired. API Gateway performs strict certificate validation during the SSL handshake.

c) TLS version compatibility: Your tests show negotiations happening with both TLSv1.2 and TLSv1.3. Ensure that your client supports both versions consistently. API Gateway now supports TLS 1.3, which offers improved performance and security.

d) Client certificate rotation: If you're using client certificates for mTLS, ensure they are rotated before expiration. API Gateway-generated client certificates are valid for 365 days and need to be rotated to avoid downtime.

e) Error handling: Implement proper error handling in your Lambda function to catch and format errors correctly. This can help provide more meaningful error responses instead of generic 500 or 502 errors from API Gateway.

f) Retry mechanism: Implement a retry mechanism in your client code to handle temporary connection issues. This can help mitigate intermittent SSL negotiation failures.

g) Check for certificate conflicts: Ensure there are no conflicts between multiple certificates with the same subject but different issuers. This can cause issues with mTLS setups.

h) API Gateway settings: Review your API Gateway settings, particularly those related to SSL/TLS and client certificate handling. Ensure they are configured correctly for your use case.

i) CloudWatch Logs: Enable detailed CloudWatch logging for your API Gateway to gain more insights into the specific SSL/TLS errors occurring during these failed connections.

By systematically addressing these points, you should be able to identify the root cause of the inconsistent behavior and either resolve it or implement a workaround to bypass the errors you're experiencing.
Sources
Troubleshoot API Gateway errors with backend SSL certificates | AWS re:Post
Handling Lambda errors with an API Gateway API - AWS Lambda
API Gateway now supports TLS 1.3
Resolve mTLS certificate subject conflicts in API Gateway | AWS re:Post
Generate and configure an SSL certificate for backend authentication in API Gateway - Amazon API Gateway

profile picture
answered 11 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions