Lambda function can access AmazonMq broker but trigger can't

Question

Hello!

I have a setup with an AmazonMq broker and 2 lambda functions that have similar configuration and access, one for producing messages (Producer) and the other to consume them (Worker).
My producer connects directly to the broker using Pika python lib and credencials from secret manager and can produce messages into my default queue without issues. I can see messages pile up in RabbitMQ web console.

On the other hand my Worker has a trigger setup but is not consuming messages, actually the Worker is never triggered and the trigger shows error:
"PROBLEM: Event source not reachable. This could mean that event source is not in an active state or Lambda reached the connection limit. Lambda will keep trying to connect with your event source."
However, if I try to connect to the broker though Pika client it works.

Here is the Event source trigger on Worker config:
* Activate trigger: Yes
* Authentication: BASIC_AUTH
* Batch size: 1
* Event source mapping ARN: arn:aws:lambda:eu-west-1:.......
* Last processing result: PROBLEM: Event source not reachable. This could mean that event source is not in an active state or Lambda reached the connection limit. Lambda will keep trying to connect with your event source.
* On-failure destination: None
* Queue name: MY_QUEUE
* Secrets Manager key: arn:aws:secretsmanager:eu-west-1:...

Worker config:
* VPC: same as broker
* Subnets: 2 PRIVATE_WITH_EGRESS subnets (1 same as broker) (Allow IPv6 traffic = false)
* Security group: No inbound rules. All outbound allowed. 
* Permissions:
* * On Secret manager: GetSecretValue
* * On AmazonMQ (Broker arn): DescribeBroker
* * On Ec2: (all resources): CreateNetworkInterface, DeleteNetworkInterface, DescribeNetworkInterfaces, DescribeSecurityGroups, DescribeSubnets, DescribeVpcs, AssignPrivateIpAddresses, UnassignPrivateIpAddresses.

Broker config:
VPC: same as broker
Subnets: 1 PRIVATE_WITH_EGRESS subnet
Security group: Allow inbound from Worker security group on port 5671.

I'm not sure what could be the next steps to debug my Event source trigger error?

Thanks!

AWSomeCSS · Answer

Based on the information provided, here are several areas to check and debug:

1. Security Group Configuration:
   - Ensure the Lambda Worker's security group allows outbound traffic specifically to port 5671 (AMQP with TLS)
   - Verify the AmazonMQ broker's security group allows inbound traffic from the Lambda's security group on port 5671

2. Network Configuration:
   - Confirm that both subnets used by the Lambda function have a route to the Amazon MQ broker
   - Check if you have NAT Gateway configured for the private subnets if needed
   - Verify that the VPC endpoints are properly configured

3. Broker Endpoint:
   - Make sure you're using the SSL-enabled endpoint (port 5671) rather than the non-SSL endpoint
   - Verify that the broker is in "Running" state

4. IAM Permissions:
   - Add `mq:CreateEventSource` and `mq:GetBrokerInstances` permissions to the Lambda execution role
   - Ensure the Lambda role has permissions to access the VPC resources

5. Event Source Mapping:
   - Try deleting and recreating the event source mapping
   - Verify the queue name is exactly correct (case-sensitive)
   - Check if the credentials in Secrets Manager are correct and in the right format

6. Debugging Steps:
   ```
   # Check broker connectivity
   aws mq describe-broker --broker-id your-broker-id

# Check event source mapping status
   aws lambda get-event-source-mapping --uuid your-mapping-uuid

# Check CloudWatch Logs for the Lambda function
   ```

7. CloudWatch Metrics:
   - Monitor the Lambda function's CloudWatch metrics for errors
   - Check AmazonMQ CloudWatch metrics for connection issues

Since you mentioned that direct Pika connection works but the event source mapping doesn't, this suggests either:
1. A permissions issue with the event source mapping
2. A network configuration issue specific to the Lambda trigger
3. An issue with the credentials format in Secrets Manager

Lambda function can access AmazonMq broker but trigger can't

Add your answer

Relevant content