By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Kafka Lambda Trigger (EventSourceMapping) - Credentials in secret not refreshed

0

Hello, I am integrating my custom Kafka cluster with AWS Lambda using Kafka trigger (Cloudformation EventSourceMapping).

If I reset the password on my Kafka cluster and subsequently update the secret that is used in the EventSourceMapping, the EventSourceMapping does not refetch/refresh the secret.

Instead the EventSourceMapping is still trying to use the old secret version (with old password) which results in error:

Last processing result: PROBLEM: SASL authentication failed.

As a result of this problem, no events are delivered to the lambda function.

Since I am passing the ARN of the secret to the EventSourceMapping and am granting Lambda the permissions to get this secret, I would expect that the EventSourceMapping would automatically refresh/refetch the secret periodically.

Instead what I need to do is delete the EventSourceMapping (trigger) and recreate it again (with the same configuration). This is unacceptable as we are using multiple Lambda integrations with our Kafka cluster and deleting/re-creating all of them after the password (and aforementioned secret) has changed is ineffective

Topics
ServerlessComputeManagement & GovernanceMicroservicesSecurity, Identity, & Compliance
Tags
AWS LambdaAWS CloudFormationMicroservicesAWS Secrets Manager
Language
English
simon
asked 2 years ago1451 views
1 Answer
0
Accepted Answer

Hello,

I understand that you are using custom Kafka event source mapping with Lambda. You did reset on existing password of Kafka cluster by associating a new Secret to the cluster. Following this, you updated Lambda event source mapping with a new Secret. However, Lambda event source seems to be still using previous secret and not doing automatic refetch/refresh on new secret, resulting in SASL authentication failure error. Please correct me if I am misunderstanding anything.

I tried to replicate this in my AWS account, where I created a new secret, associated new secret to Kafka cluster. Further, I updated the Kafka client username/password to match the values in the new secret. Finally, I updated Lambda event source mapping to use new Secret ARN. Here is the CLI command I used to update the event source:

$ aws lambda update-event-source-mapping --uuid <Event Source Mapping ID> --function-name <Function_Name> --source-access-configurations "Type"="SASL_SCRAM_512_AUTH","URI"="<New_Secret_ARN"

After the update, I was able to successfully authenticate using new secret credentials, and deliver Kafka events to Lambda. There is no need to delete and re-create Lambda Event Source.

In order to further troubleshoot your specific issue, please specify whether you are using CloudFormation to deploy all changes and update. Assuming you are using CloudFormation to update Lambda with new secret, do you see the new Secret ARN being reflected in Console or CLI (list-event-source-mappings). Do you see the error "Last processing result: PROBLEM: SASL authentication failed.", in Lambda logs. If possible, please run a test for Lambda event source update, using AWS CLI and confirm if the issue persists.

Related Links:

AWS
SUPPORT ENGINEER
Isha_K
answered 2 years ago
  • simon
    2 years ago

    Actually I was not updating the event source mapping, I was updating the secret itself. However, I have already resolved this problem.

    It seams that there was short period of time (after I updated the secret), that the event source mapping was trying to still use the old credentials. After waiting a while event source mapping refreshed the value of the secret and it started working.

    Thanks for your help

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content