Aurora Serverless Secret issue (Connection failed/ Access Denied for User)

Hi, I wasn't able to find a thread answering this issue (or any advice online), so I'll start a discussion and hope some of you may have advice. I'm not sure if I should be posting this in the Secrets Manager forum, or RDS one.

I've created Aurora Serverless database cluster. Any secret which I've created to interact with this cluster fails to connect to it.

For some context, I'm creating the cluster, secret (and stack in general) through the AWS Cloud Development Kit. I will provide the output CloudFormation template below. I can provide my code is that's helpful.

Resources:
  <clusterid>:
    Type: AWS::RDS::DBCluster
    Properties:
      Engine: aurora
      DatabaseName: DBName
      DBClusterIdentifier: <clusteridentifierid>
      EngineMode: serverless
      MasterUsername: root
      MasterUserPassword: notpassword
    Metadata:
      aws:cdk:path: SecretTestStackID/<clusterid>
  SecretTSID:
    Type: AWS::SecretsManager::Secret
    Properties:
      GenerateSecretString:
        ExcludeCharacters: '"@/\'
        GenerateStringKey: password
        PasswordLength: 16
        SecretStringTemplate: '{"username": "test"}'
      Name: SecretTSName
    Metadata:
      aws:cdk:path: SecretTestStackID/SecretTSID
  SecretTargetAttachmentTSID:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId:
        Ref: SecretTSID
      TargetId:
        Fn::Join:
          - ""
          - - "arn:aws:rds:eu-west-1:"
            - Ref: AWS::AccountId
            - ":cluster:"
            - Ref: <clusterid>
      TargetType: AWS::RDS::DBCluster
    Metadata:
      aws:cdk:path: SecretTestStackID/SecretTargetAttachmentTSID

I came across this problem whilst trying to make use of the new Data API for Aurora Serverless. First of all, I deployed all my resources through CDK. I have a custom resource which modifies my DB cluster to enable HTTP endpoints (making the Data API usable). I can verify the modification is successful through viewing the cluster in the AWS console. However, my attempt to use the Data API fails when I execute the beginTransactionRequest - where my ClusterARN, SecretARN and DatabaseName are provided to the DataAPI.

It would log the error below. However, I think this is log is incorrect, as I noted before, I can verify through the UI that the HTTP endpoints have been enabled through the console:
INFO: {rds-data, eu-west-1} was not found in region metadata, trying to construct an endpoint using the standard pattern for this region: 'rds-data.eu-west-1.amazonaws.com'.
com.amazonaws.services.rdsdata.model.BadRequestException: HttpEndpoint is not enabled for cluster <ClusterIdentifierID>. Please refer to https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html#data-api.troubleshooting

I'm making use of the Data API through a Lambda function, and I checked the IAM policy for the Lambda, and it appears to have the requisite actions/ resources.

I figured I'd check to see if there were any problems which each of those inputs. The ClusterARN is valid (afaik) because I'm using it prior to this transaction request in my custom resource, where I modify the cluster. I know the DatabaseName is valid, because it's the same string I used as an argument for it. The SecretARN appears to be the issue. However, I can verify that the correct ARN is being passed as an argument, as I can view it in the SecretsManager part of the console.

I decided to try manually use the generated secret in the RDS console, with the Query console. However, when I entered the credentials to log in there (from the SecretsManager console), I recieve an error telling me that my Connection failed.

I thought perhaps I was making a mistake whilst creating my secret through CDK. I tried to create a secret manually (through the SecretsManager console), associating it with my RDS cluster. However, the same problem occurs when I try to use the RDS Query console.

I should note, when creating the cluster in CDK, I provide it a master username & password. I'm able to log into the RDS Query console and interact with my cluster successfully with these credentials.

Is there something I appear to have missed when trying to connect things together? Unfortunately the error messages I'm getting from AWS don't explain why the secret isn't working - I'm hoping someone might be able to point me in the right direction as to why. I provided the CloudFormation output relating to my cluster, the secret and the secret attachment, but I can provide the CDK code too, if necessary.

Edited by: markmcs on Jun 21, 2019 7:18 AM
Edit: I will also note, I saw some people noting they had issues logging into their RDS instances due to the settings on their parameter group. The advice was to create a new parameter group and set 'log_bin_trust_function_creators' to 1. I tried this (modifying my cluster to use this new group instead) but it made no discernible impact.

Edited by: markmcs on Jun 21, 2019 10:21 AM
I just created an Aurora Serverless cluster through the AWS console. There were few (seemingly relevant) options to configure. I then created another secret to test through the console. My cluster has been created fine, but the secret still doesn't work.

Edited by: markmcs on Jun 21, 2019 10:23 AM