PKIX path building failed

I ended up adding a prebuild sript that uses letsencrypt and the route 53 plugin. I couldn't get the api gateway to accept the self signed certs I created, so in the interest of time, I went with this solution since it fixed the issue the fastest. The code is below for anyone who is interested.

#!/bin/bash

domain='api.example.com'
contact='email here'
bucket='s3://elasticbeanstalk-us-west-2-accountid'
bucket_path="${bucket}/letsencrypt"

# Add cron job
function add_cron_job {
    touch /etc/cron.d/certbot_renew
    echo "* * * * * webapp 0 2 * * * certbot renew --allow-subset-of-names
    # empty line" | tee /etc/cron.d/certbot_renew
}

#check if certbot is already installed
if command -v certbot &>/dev/null; then
    echo "certbot already installed"
else
    # Install certbot and plugin since it's not installed already
    # Instructions from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SSL-on-amazon-linux-2.html#letsencrypt

    sudo dnf install -y python3-certbot-nginx
    sudo dnf install -y python3-certbot-dns-route53
fi

# check if the S3 bucket already exists with a certificate
if [ -n "$(aws s3 ls $bucket_path)" ]; then

    # download and install certificate from existing S3 bucket
    echo "$bucket_path exists."
    sudo rm -rf /etc/letsencrypt/*
    sudo aws s3 cp ${bucket_path}/backup.tar.gz /tmp
    sudo tar -xzvf /tmp/backup.tar.gz --directory /
    sudo chown -R root:root /etc/letsencrypt

    # -n: Non-interactive mode (runs without prompting the user for input).
    # -d ${domain}: Specifies the domain(s) for the certificate.
    # --nginx: Automatically configures Nginx with the certificate.
    # --agree-tos: Automatically agrees to the terms of service.
    # --email ${contact}: Sets the email for renewal notifications.
    # --reinstall: Reinstalls an existing certificate.
    # --redirect: Forces HTTPS.
    # --expand: Adds domains to an existing certificate.
    # --allow-subset-of-names: Issues a certificate even if some domains fail validation.
    sudo certbot -n -d ${domain} --nginx --agree-tos --email ${contact} --reinstall --redirect --expand --allow-subset-of-names
    systemctl reload nginx

    # re-uploading the certificate in case of renewal during certbot installation
    tar -czvf /tmp/backup.tar.gz /etc/letsencrypt/*
    aws s3 cp /tmp/backup.tar.gz ${bucket_path}/backup.tar.gz

    add_cron_job
    exit
fi

# obtain, install, and upload certificate to S3 bucket since it does not exist already
sudo certbot certonly -n -d $domain --preferred-challenges dns --dns-route53 --email ${contact} --agree-tos
tar -czvf /tmp/backup.tar.gz /etc/letsencrypt/*
aws s3 cp /tmp/backup.tar.gz ${bucket_path}/backup.tar.gz

add_cron_job

Greeting

Hi Spencer!

Thanks for reaching out with your question. Let’s get to the bottom of this PKIX path building issue and find a solution together. 😊

---

Clarifying the Issue

You’re working with an Elastic Beanstalk environment running behind an AWS API Gateway. Your setup uses end-to-end encryption with a Network Load Balancer (NLB) and relies on a self-signed certificate for the EC2 instances. Despite following the documentation to bypass SSL verification in the API Gateway, you’re encountering a persistent error:

Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

You’ve confirmed that the API is reachable via a jump server using curl, which suggests the issue lies with the API Gateway configuration or its trust of the self-signed certificate.

---

Key Terms

• PKIX (Public Key Infrastructure X.509): A standard for managing digital certificates and public-key encryption.
• PKIX Path Building Error: This occurs when a certificate presented during an SSL/TLS handshake cannot be validated because the chain of trust to a recognized root certificate is incomplete.
• Self-Signed Certificate: A certificate that is not issued by a trusted Certificate Authority (CA) and often requires additional configuration for secure communication.
• Elastic Beanstalk: A platform to deploy and scale web applications and services.
• AWS API Gateway: A managed service to create, publish, and secure APIs.

---

The Solution (Our Recipe)

Steps at a Glance:

1. Understand why PKIX errors occur and determine if self-signed certificates are necessary for your setup.
2. Configure your API Gateway to trust your self-signed certificate by adding it to a custom trust store.
3. Verify that the NLB listeners are correctly configured to forward HTTPS traffic to your instances.
4. Enable mutual TLS (mTLS) in the API Gateway if needed for additional security.
5. Check CloudWatch logs and enable debug logging for API Gateway to pinpoint any remaining issues.
6. Test the configuration using curl and Postman to validate the API endpoint.

---

Step-by-Step Guide:

1. Understand PKIX Path Errors and Consider Alternatives:
   • PKIX path errors occur when a client (in this case, API Gateway) cannot verify the authenticity of a server's certificate because it is self-signed or not part of a trusted chain.
   • Best Practice: For production environments, consider obtaining a certificate from a recognized Certificate Authority (CA). AWS ACM (AWS Certificate Manager) offers free public certificates that integrate seamlessly with NLB and API Gateway.

---

2. Configure API Gateway Trust Store:
   • First, export your self-signed certificate’s public key in .pem format:

     openssl x509 -in <your_certificate.crt> -outform PEM -out certificate.pem

     This ensures the API Gateway can use the certificate to validate your backend instances.
   • Navigate to API Gateway in the AWS Management Console:
     1. Open the Custom Domain Names section in the left-hand navigation pane.
     2. Click on your domain name or create a new one.
     3. Scroll down to Mutual TLS Settings and upload your .pem file to create a trust store.
   • Associate this trust store with your API Gateway, ensuring that the backend's self-signed certificate can be trusted during SSL/TLS validation.

---

3. Verify NLB Listener Configuration:
   • In the EC2 Management Console, ensure the NLB listener forwards HTTPS requests to your EC2 instances on the correct port.
   • Confirm that the health checks are configured to use HTTPS as the protocol to validate instance readiness.

---

4. Enable Mutual TLS (Optional):
   • If your architecture requires client-side certificate validation, configure mutual TLS using the AWS CLI:

     aws apigatewayv2 update-domain-name --domain-name <your_domain_name> --mutual-tls-authentication truststoreUri=<s3_uri> --truststoreVersion=<version_id>

   • Replace <your_domain_name>, <s3_uri>, and <version_id> with your specific values.

---

5. Check CloudWatch Logs for Insights:
   • In the API Gateway Console, enable detailed logging for your stage:
     1. Navigate to Stages under your API.
     2. Choose a stage and enable CloudWatch Logs.
     3. Set the logging level to Debug.
   • Review logs in the CloudWatch Console for any clues about configuration issues.

---

6. Test the Configuration:
   • Using curl:

     curl -k -X GET https://<api_gateway_url>/<endpoint> --header "Authorization: Bearer <token>"

     The -k flag bypasses SSL validation, which is useful for testing self-signed certificates.
   • Using Postman:
     1. Open Postman and create a new request.
     2. Set the method to GET and enter your API Gateway URL (https://<api_gateway_url>/<endpoint>).
     3. In the Authorization tab, add your Bearer token if required.
     4. Under Settings (the gear icon in the top-right corner), turn off SSL certificate verification.
     5. Send the request and verify the response.

---

Closing Thoughts

Configuring self-signed certificates with AWS services requires extra attention to trust store configurations and endpoint validation. However, using CA-signed certificates for production systems eliminates many of these challenges and simplifies operations. If you’re testing or in development mode, the steps provided should help you resolve this PKIX path error.

If you encounter further issues or decide to switch to ACM-provided certificates, feel free to ask for more guidance. Good luck with your setup, and I’m here if you need more help!

---

Farewell

Best wishes, Spencer! Let us know how it goes. Have an awesome day! 😊

---

Cheers,

Aaron 🚀✨