Integrating External PKI with AWS Private CA and cert-manager for Dynamic Cert Management in EKS

In this article, we’ll explore how you can integrate your external PKI with AWS Private Certificate Authority (PCA) and the open-source cert-manager project to enable dynamic certificate management for your workloads running on Amazon Elastic Kubernetes Service (EKS).

Envision a scenario where you possess an established on-premises PKI infrastructure lacking native Kubernetes integration. However, you desire to retain its usage while simultaneously harnessing the dynamic capabilities of programmatically managing certificates through Kubernetes.

Architecture

In the following steps, we’ll establish a basic root certificate that will serve as our fictional external PKI infrastructure. This certificate will facilitate trust between the fictional root CA and AWS PCA. Subsequently, we’ll configure cert-manager to establish direct communication with PCA, enabling it to manage certificates for applications within our EKS cluster as required.

Creating our root CA

This step is straight forward, we will create the root certificate using the following commands:

# Let's start by creating our private and public keys for our external root CA.
# If you skip the `-subj` argument, you will be prompted to fill out such details
# interactively. When prompted for passphrase, enter one, and remember it for the
# rest of this tutorial.
$ openssl req -new -x509 -days 3650 -extensions v3_ca -keyout root_ca_privatekey.pem \
  -out root_ca_cert.pem \
  -subj "/CN=Octank Inc Root CA/C=SE/ST=Stockholm/L=Stockholm/O=Octank"

Configure AWS Private Certificate Authority

Now that we have successfully created our root CA, we will proceed to configure AWS PCA.

1. Start by going to the AWS PCA console, and click Create a private CA.
2. For Mode options, select General-purpose.
3. Specify CA type: Subordinate.
4. Fill other details as appropriate

Install subordinate certificate in AWS PCA

Next, we’ll create and sign a certificate for our subordinate CA.

1. Navigate to the Certificate Authority we just created in the AWS PCA console.
2. Click Actions→Install CA Certificate.
3. Select External private CA as the CA type
4. This will generate a certificate signing request (CSR), export this CSR to a file, call the file AWS_PCA_CSR.pem.

Now, we can sign the certificate request using the root certificate we previously created:

# Sign the certificate using our root cert. Note that we are specifying `-copy_extensions copyall`
# here to ensure that we bring over the basic constraints specified by the
# certificate request which are needed by AWS PCA.
$ openssl x509 -req -in AWS_PCA_CSR.pem -CA root_ca_cert.pem -CAform PEM \
  -CAkey root_ca_privatekey.pem -out AWS_PCA_signed.pem -copy_extensions copyall

Once we have signed the CSR, the command above will output a file called AWS_PCA_signed.pem. Let’s continue the configuration of our PCA:

1. Upload AWS_PCA_signed.pem file in the field called certificate in the PCA console.
2. Given we don’t have a longer chain of trust we need to consider for our purposes in this article, we can simply upload our root_ca_cert.pem in the certificate chain field in the PCA console.

Configure cert-manager

This article does not provide detailed instructions on installing cert-manager or creating an EKS cluster for this purpose. We assume you followed the Helm installation guide for cert-manager and have successfully created an EKS cluster using eksctl.

Install and configure Issuer for AWS PCA

Next, we are going to install the issuer for cert-manager that interacts with AWS Private CA. You can find details about the issuer in this GitHub repository. We will first create a ServiceAccount which is connect to an AWS IAM Role, so that the PCA Issuer can assume that role and talk to the AWS PCA APIs to manage certificates for us. In this article we will use a predefined IAM policy. NOTE: In your production environment you want to use a more narrowly scoped down IAM policy which only allows access to the ARN of your specific CA.

While in this guide, we will use IAM Roles for Service Accounts (IRSA) to scope IAM permissions to the pods that have access to the ServiceAccount we will create below. Recent versions of cert-manager and the AWS PCA issuer should both support EKS Pod Identities to achieve the same functionality.

$ eksctl create iamserviceaccount -c [CLUSTER_NAME] \
  --namespace=cert-manager --name=awspcaissuer \
  --attach-policy-arn=arn:aws:iam::aws:policy/AWSPrivateCAFullAccess \
  --approve

Once we have configured the ServiceAccount, we can install the AWS PCA issuer using Helm, and specifying which ServiceAccount to use:

$ helm -n cert-manager install \
    --namespace=cert-manager \
   --set serviceAccount.create=false \
   --set serviceAccount.name=awspcaissuer awspcaissuer awspca/aws-privateca-issuer

We should now be able to see pods running for the AWS PCA issuer in the cert-manager namespace containing a serviceAccount specification matching the name we created in the previous step:

$ kubectl -n cert-manager get pod awspcaissuer-aws-privateca-issuer-6b45665dbd-crt5h -o yaml|grep 'serviceAccount:'
  serviceAccount: awspcaissuer

Now we can create an issuer in the cluster for our AWS PCA certificate authority which we created previously:

$ kubectl -n cert-manager apply -f - <<EOF
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
  name: subordinate-ca-issuer
spec:
  arn: arn:aws:acm-pca:$AWS_REGION:$ACCOUNT_ID:certificate-authority/[ID_OF_CA]
  region: $AWS_REGION
EOF

Using our CA from Kubernetes

And finally, what we have been waiting for, creating a certificate using Kubernetes APIs via cert-manager, which we can use with our applications:

$ kubectl -n default apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: demo-cert
  namespace: default
spec:
  commonName: demo.engineering.octank.com
  dnsNames:
    - demo.engineering.octank.com
  duration: 168h0m0s
  issuerRef:
    group: awspca.cert-manager.io
    kind: AWSPCAClusterIssuer
    name: subordinate-ca-issuer
  renewBefore: 160h0m0s
  secretName: demo-cert-secret
  usages:
    - server auth
    - client auth
  privateKey:
    algorithm: "RSA"
    size: 2048
EOF

Note: You must specify the duration and renewals that conform to the details of your subordinate Certificate Authority (CA). The specifics for your certificate request may vary, and this is merely an example.

We can now have a look at the certificate as it is represented in Kubernetes, both as a separate Certificate object, which contains metadata for the certificate, and as a Secret, which holds the actual certificate data:

$ kubectl -n default get certificate demo-cert -o yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  [..snip..]
spec:
  [..snip..]
status:
  conditions:
  - lastTransitionTime: "2023-12-18T16:40:04Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 2
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2023-12-25T16:40:01Z"
  notBefore: "2023-12-18T15:40:01Z"
  renewalTime: "2023-12-19T00:40:01Z"
  revision: 1

$ kubectl -n default get secret demo-cert-secret -o yaml
apiVersion: v1
data:
  ca.crt: [..snip..]
  tls.crt: [..snip..]
  tls.key: [..snip..]
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: demo.engineering.octank.com
    cert-manager.io/certificate-name: demo-cert
    cert-manager.io/common-name: demo.engineering.octank.com
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: awspca.cert-manager.io
    cert-manager.io/issuer-kind: AWSPCAClusterIssuer
    cert-manager.io/issuer-name: subordinate-ca-issuer
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2023-12-18T16:40:04Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: demo-cert-secret
  namespace: default
  resourceVersion: "340115576"
  uid: b772114a-dac9-40cd-ad1b-73d57b1dce59
type: kubernetes.io/tls

The Secret can now be mounted inside pods that need access to the certificate, so that it can be used by your applications.

Now, we can validate our issued certificate against the chain of trust we’ve created between our root CA and subordinate CA. First, we need to create a file containing our certificate bundle, we need to grab the certificate from our AWS PCA CA:

$ aws acm-pca get-certificate-authority-certificate \
    --certificate-authority-arn [ARN_OF_YOUR_CA] \
    --output text \
    --query 'Certificate' > subordinate_ca.pem

and create a file containing our bundle:

$ cat subordinate_ca.pem root_ca_cert.pem > certbundle.pem

finally, let’s grab the certificate data from the Kubernetes Secret and validate it against our cert bundle:

# Get the tls.crt part of the Kubernetes secret
$ kubectl -n default get secret demo-cert-secret -o jsonpath="{.data['tls\.crt']}"|base64 -d > issued_cert.pem

# Validate the certificate against the previously created cert bundle containing both
# the root and intermediate CA
$ openssl verify -CAfile certbundle.pem issued_cert.pem
issued_cert.pem: OK

Now that we have verified the issued certificate against the certificate bundle containing our root and subordinate CA public certificates, it's important to note that this certificate bundle would need to be distributed to any clients that need to trust the certificates issued by our integrated PKI and AWS PCA setup. Without the full chain of trust, clients would be unable to validate the authenticity of the certificates presented to them by services running in the EKS cluster. Therefore, in a production environment, you would need to ensure that the certificate bundle, containing the public root CA certificate and any public intermediate CA certificates, is installed on all relevant client systems that need to communicate with your Kubernetes applications. The private CA certificates would be securely managed within your PKI infrastructure and the AWS PCA service, and would not be included in the distributed certificate bundle. This distribution of the public certificate bundle is a crucial step to complete the trust relationship and allow clients to reliably authenticate the services in your cluster. Only after the certificate bundle is in place can you fully leverage the dynamic certificate management capabilities provided by the integration of your external PKI, AWS PCA, and cert-manager.

Conclusion

In this guide, we have explored how you can integrate your on-premises or external certificate authorities with AWS PCA as an intermediate CA, so that you can fully leverage the dynamic nature of certificate provisioning through cert-manager.