2 Answers
- Newest
- Most votes
- Most comments
2
There is a native retry functionality within Glue through the MaxRetries parameter. This parameter can be defined programmatically or if using Glue Studio in the "Job details" tab.
MaxRetries – Number (integer). The maximum number of times to retry this job after a JobRun fails.
If this is not sufficient for your use-case, consider wrapping your Glue job within a Step Function and here you can implement a more sophisticated retry/control mechanism.
answered 2 years ago
0
Thanks for you answer, i'm gonna test the suggest steps and let you know :)
answered 2 years ago
Relevant content
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?
AWS OFFICIALUpdated 2 years ago - AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated a year ago
Hi, We have a Glue job that accepts the parameter "table_name," with the default value set as "dummy" in the Glue job parameters section. Additionally, the Glue job configuration allows a total of 4 retries.
A Lambda function was created to invoke the Glue job with the parameter "table_name" set to "xxxx." During the initial run, the Glue job failed with the error message: "The specified subnet does not have enough free addresses to satisfy the request. Please provide a connection with a subnet with IPs available" and "exception: Number of IP addresses on subnet is 0."
Despite these errors, the Glue job retried itself 4 times as per the configuration. However, in each retry attempt, the "table_name" parameter inexplicably changed to the default value "dummy," deviating from the original value passed by the Lambda function ("xxxx"). This unexpected parameter change caused the Glue job to fail repeatedly with error "An error occurred while calling o93.getCatalogSource. : com.amazonaws.services.glue.model.EntityNotFoundException: Entity Not Found".
Could you please confirm why the parameters in the Glue job are reverting to default values when the job is retried
+1 to the above comment, In my case the job failed due to a connection issue with redshift, which defaulted to automatic retires, where the default parameters which were passed to the glue job were missing in the job instances that were being triggered due to original job run failure.