CloudHSM KSP 5.15.0 RSA-PSS incompatibility with Windows Authenticode/MSIX signing

Based on the information available, there are a few known issues with AWS CloudHSM Client SDK versions, but none specifically addressing the RSA-PSS padding issue you're experiencing with MSIX package signing.

For your specific questions:

1. The documentation doesn't mention a direct configuration option to force PKCS#1 v1.5 padding for Authenticode signing when using the KSP. The registry configuration you tried (HKLM:\SOFTWARE\Cavium\Crypto\RSASignaturePadding=1) seems like a reasonable approach, but it's not explicitly documented as a solution for this issue.

2. While there are newer versions of the CloudHSM Client SDK (such as 5.16.1 and 5.16.2) that fix other issues, there's no specific mention of fixing RSA-PSS padding incompatibility with Windows Authenticode/MSIX signing. However, upgrading to the latest version might be worth trying as it could include undocumented fixes.

3. As a workaround, you might consider using the key_mgmt_util (KMU) for signing operations instead of the KSP. The KMU allows you to explicitly specify the signing mechanism. For example, you can use the "sign" command with the SHA256_RSA_PKCS mechanism (represented by integer 1) which uses PKCS#1 v1.5 padding. This would require adjusting your signing workflow to use the KMU directly rather than SignTool.

Another potential approach would be to validate if there are any SignTool command-line parameters that can explicitly specify the padding method to override the default behavior when working with the CloudHSM KSP.

If these approaches don't work, you might need to reach out to AWS Support for a more specific solution to this particular compatibility issue.
Sources
Known issues for the Key Storage Provider (KSP) for AWS CloudHSM - AWS CloudHSM
Known issues for all HSM instances - AWS CloudHSM
Generate a signature using AWS CloudHSM KMU - AWS CloudHSM