Best way of establishing authentication between AWS Lambda and MSK for producing messages

Hi,

To handle authentication between AWS Lambda and MSK for producing messages, particularly when multiple SQS Queues trigger Lambda instances that need to produce to MSK topics, you have two options:

For IAM Access Control, while it's the simpler option to implement, there's a significant limitation: MSK has a quota of maximum 100 TCP connections per broker when using IAM authentication. This can become problematic when multiple Lambda instances (producer side) initiate connection requests, potentially exceeding this limit during high-load scenarios.

The alternative is mTLS-based authentication, which doesn't have the connection limit constraint but requires careful consideration of potential latency during the initial TLS handshake. However, this latency is typically minimal and can be mitigated through connection pooling strategies.

Given these considerations, here's the recommended approach: If your workload consistently requires more than 100 concurrent Lambda instances producing messages, use mTLS authentication with a connection pooling implementation. This provides better scalability and removes the IAM connection limit constraint. However, if your workload is smaller and you prefer easier implementation, IAM authentication is sufficient.

For either approach, ensure proper error handling and monitoring. With mTLS, implement certificate rotation and management procedures. With IAM, monitor connection counts and implement backoff strategies if approaching limits.

For implementation details, refer to:

• MSK Authentication Documentation
• Lambda VPC Configuration
• MSK Quotas and Limits