AWS Glue Workflow Not Saving Job Parameters

0

Hello,

I am trying to save job parameters that I would like to pass to individual jobs within the workflow. After adding them and clicking the 'Update' button and reopening the job parameters, all of the parameters disappear. Could someone advise on how to do fix this issue or an alternate way to pass parameters to several glue jobs and trigger crawlers on a schedule? They are custom parameters that I am adding to parse through via the argument parser. For example, I add a parameter to identify the primary key as an input for a job as well as read and write paths.

  • What are the parameters, can you please mention those here.

Nick
질문됨 10달 전279회 조회
1개 답변
0

Hello,

I understand that you are trying to save job parameters to pass to individual jobs within the workflow. But after adding them and reopening the job parameters, they do not appear on the console.

To rule out any permission issues, could you please verify if you have the below required permissions to update the workflow run properties.

UpdateWorkflow action - Updates an existing workflow.
PutWorkflowRunProperties action - Puts the specified workflow run properties for the given workflow run.
GetWorkflowRunProperties action - Retrieves the workflow run properties which were set during the run.

You can also check your cloudtaril logs to see if there are any errors while saving the parameters.

Please refer below documentation for more details:
[+] https://docs.aws.amazon.com/glue/latest/dg/workflow-run-properties-code.html
[+] https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-workflow.html#aws-glue-api-workflow-PutWorkflowRunProperties
[+] https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-workflow.html#aws-glue-api-workflow-UpdateWorkflow

Also, please try clearing your browser cache and try again and use a different browser once to see if that helps.

Additionally, you can use the get-workflow-run-properties(https://docs.aws.amazon.com/cli/latest/reference/glue/get-workflow-run-properties.html) CLI command to check if the run properties were set for the workflow or not.

AWS
Ankur_J
답변함 10달 전
  • I have admin privileges and this issue is persisting through several browsers over a multi-day period. I receive a green bar at the top that indicates my parameters were saved but upon opening and refreshing, the parameters still disappear. I reference these parameters to create other workflow jobs as well, so the GUI is helpful in that respect.

  • Additionally, some of my jobs are now failing because they parameters are not being passed to the jobs

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠