A fargate task schedule via EventBridge fails to launch sometimes with stopReason "Rate limit exceeded while preparing network interface to be attached to instance"

Question

Hello all,

Our environment contains:
- AWS EventBridge event to run an AWS Fargate task once an hour;
- AWS ECS event and lambda to monitor task status changes

Lately, we sometimes see that the AWS Fargate task fails to start due to the stopReason:

> Rate limit exceeded while preparing network interface to be attached to instance

I have checked our quotas, and we are very far (<50 ENIs out of 5000 per region).

1. What is the cause for this error?
2. Is a retry (the lambda to RunTask the task) a good workaround?

Thank you in advance,
Boris.

Answer

It sounds as though perhaps there's something in your account/region, unknown to you so far, that is manipulating ENIs frequently. I would recommend enabling CloudTrail and checking the activity logs for EC2 AttachNetworkInterface actions. A high request rate could result in the symptom you're describing.

A fargate task schedule via EventBridge fails to launch sometimes with stopReason "Rate limit exceeded while preparing network interface to be attached to instance"

Relevanter Inhalt