Using MSK as trigger to a Lambda with SASL/SCRAM Authentication
Hi, I have set up a MSK cluster with SASL/SCRAM authentication. I have stored the username and password in a secret using AWS Secrets Manager. Now I am trying to set the topic in the MSK cluster as an event source to a Lambda function. In order to do so, I am following this documentation: https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/ However the above documentation is for unauthenticated protocol. So I tried to add the authentication and the secret. I also added a policy in the execution role of the Lambda function that lets it read the secret value:
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:*"
],
"Resource": [
"arn:aws:secretsmanager:****:*******:secret:*"
]
},
{
"Effect": "Allow",
"Action": "secretsmanager:ListSecrets",
"Resource": "*"
}
]}
When I am trying to add the trigger, I am getting the error: An error occurred when creating the trigger:
Cannot access secret manager value arn:aws:secretsmanager:::secret:*******. Please ensure the role can perform the 'secretsmanager:GetSecretValue' action on your broker in IAM. (Service: AWSLambda; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: ****; Proxy: null)
I am not able to understand this error since I have included in the policy all the Actions from "secretsmanager" on all the Resources in my account. Can someone help?
Hi, Thanks for your reply. Since I would like the MSK Cluster have public access, I cannot have clusters hosted in private subnets inside the VPC. I do have a NAT Gateway running in the Public subnet. I am able to produce and consume messages to the topic in this cluster from terminal, but when trying to redirect the message to the Lambda function as a payload, I am facing the above mentioned problem.
Since Lambda must have access to your VPC to use Amazon MSK as a trigger, you need to make sure that, if the subnets you chose are private, there is connectivity to a NAT Gateway or to a Secrets Manager VPC Endpoint to access Secrets Manager.
Relevant questions
How do you setup cross-account IAM authentication in AWS MSK?
Accepted Answerasked 4 months agoKafka Lambda Trigger (EventSourceMapping) - Credentials in secret not refreshed
Accepted Answerasked 4 months agoMSK Custom Configuration using Cloudformation
asked 4 months agoUsing MSK as trigger to a Lambda with SASL/SCRAM Authentication
asked a month agoAmazon MSK Authentication and Authorization
asked 6 months agoHow to connect Glue to MSK with IAM authentication?
asked 7 months agoLambda MSK trigger not working
asked 2 years agoMSK ebs snapshot
asked 2 years agoMSK Connect - Failed to create using in-built AWSServiceRoleForKafkaConnect role
asked 5 months agoWhat Username do AWS VPN Client need when using password-encrypted private key certificate?
asked 3 months ago
Are you deploying the Lambda function with VPC access? Is that the same VPC where the MSK cluster is running?