Step function state to execute a Glue job seems to be stalling
Hi - I have a step function set up that invokes a glue job. This happens successfully, and the glue job succeeds in about 30 seconds. The step function state simply stays on running
, never moving to the next state. I've waited about an hour after the glue job succeeded.
Here is my state for the glue job:
"Glue-Transform": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "job-name",
"Arguments": {
"--argument.$": "$"
}
},
"Catch": [{
"ErrorEquals": ["States.DataLimitExceeded",
"States.Runtime",
"States.Timeout",
"States.TaskFailed",
"States.Permissions"],
"ResultPath": "$.Error",
"Next": "MapFailed"
}],
"Catch": [{
"ErrorEquals": ["States.ALL"],
"ResultPath": "$.Error",
"Next": "MapFailed"
}],
"End": true
}
I haven't been able to find documentation on troubleshooting anything related to this, besides this page which only details how to start a simple job, which I believe I am following correctly: https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html
Found my answer here - https://stackoverflow.com/questions/56812780/how-to-have-a-python-glue-job-return-when-called-in-step-function
The solution to my actual problem was permissions. You need four permissions when running a startJogRun.sync:
glue:StartJobRun
glue:GetJobRun
glue:GetJobRuns
glue:BatchStopJobRun
Those are actually the Terraform values, but should help anybody struggling with this.
Relevant questions
AWS glue combining multiple input into a single output csv
asked 3 months agoSophisticated Triggering of Glue Jobs
asked 17 days agoStep Function action GetJob (AWS Glue) does not return CodeGenConfigurationNodes despite the documentation saying it should
asked a month agoUpdate Records with AWS Glue
asked a month agoCan Glue bookmarks be used between Glue workflow steps?
Accepted Answerasked 3 years agoStep function state to execute a Glue job seems to be stalling
asked a year agoCall a glue Job from within another without using vpc endppoints or SG
asked 3 months agoLoading json string data as super from Glue job in to Redshift
asked 5 months agoGlue Catalog API call
asked a month agoAWS Glue retry a job after an execution error
Accepted Answerasked a month ago