AWS Data Pipeline stuck in "WAITING FOR RUNNER"

Common issue would be ,

1. check the IAM access of emr ec2 role and make sure that it has datapipeline:*
2. make sure that internet connectivity is available from emr/ec2 either nat or igw

you can add ec2 key in emr cluster and log in to emr master (ssh) then check task runner log

tail -100f /mnt/taskrunner/output/logs/tasrunner*.log

https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-how-task-runner-user-managed.html#dp-verify-task-runner

Hi Shivan,
thank you for your answer.

I think with EMR EC2 Role you mean the DataPipelineDefaultResourceRole? The one i assign durin the creation of the pipeline to the field "EC2 instance role"?
If so, i checked. It has datapipeline full access.

If i activate my datapipeline, then switch to the EMR console, and check the respective cluster, it says "Install Task Runner cancelled".

Do you have an idea what the issue could be?

Best regards

Edited by: OlliHC on Jan 14, 2019 7:08 AM

Edited by: OlliHC on Jan 14, 2019 7:14 AM

check your pipeline terminateaftertimeout parameter and increase 2-3 times normal run +15 minutes for cluster launch

Example,
Job usual run time is 10 minutes then 30 +15 =45 minutes would be best value for terminateafter

Just to be precise:
I fixed it.

The reason was that our limit on EC2 instances of the respective account was reached.
We texted the AWS support and their increased our limit, now it works fine.