Behavior of Dependent Jobs when Jobs they Depend on need Automated Retry

0

I have a Lambda that submits batch jobs for 2 categories of jobs (setup jobs, and data jobs).

The data jobs are dependent on the success of setup jobs, so I get the job ids of the setup jobs and set them as a dependency for the data files. Currently, there is no auto-retry so the data jobs fail immediately once any setup job fails.

I want to have auto-retry on the setup jobs but first want to know about the expected behavior of the dependent data jobs -- If I put the data jobs in queue with a dependency on the setup jobs, will the data jobs fail at the FIRST failure? Or will the dependent data jobs stay in 'PENDING' state until all retry attempts are made on the setup jobs?

Thank you.

已提問 1 年前檢視次數 228 次
1 個回答
0

From the official documentation[1] we can see that:

PENDING: A job that resides in the queue and isn't yet able to run due to a dependency on another job or resource. After the dependencies are satisfied, the job is moved to RUNNABLE.

I have also replicated your scenario for your better understanding:

→ lets assume B is dependent on A, where A is the failing job

→ I created a job "A" which will fail on purpose by giving the a invalid command (slee 10 in this case), I have set the 'Job attempts' to 10

→ At the very same time, I have submitted another job "B" with the dependency "A"

→ I could see that the job "B" was in 'PENDING' status till all the retries were completed

→ Once all the retries had failed for job "A", job "B" goes into 'FAILED' status as well.

So Answering your query: Q. I want to have auto-retry on the setup jobs but first want to know about the expected behaviour of the dependent data jobs -- If I put the data jobs in queue with a dependency on the setup jobs, will the data jobs fail at the FIRST failure? Or will the dependent data jobs stay in 'PENDING' state until all retry attempts are made on the setup jobs?

➢ The dependent data jobs stay in 'PENDING' state until all retry attempts are made on the setup jobs!

Reference links:

[1] Job states - https://docs.aws.amazon.com/batch/latest/userguide/job_states.html
[2] Submitting a job - https://docs.aws.amazon.com/batch/latest/userguide/submit_job.html
[3] Job dependencies - https://docs.aws.amazon.com/batch/latest/userguide/job_dependencies.html?icmpid=docs_console_unmapped

AWS
已回答 1 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南