By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Glue Workflow Not Saving Job Parameters

0

Hello,

I am trying to save job parameters that I would like to pass to individual jobs within the workflow. After adding them and clicking the 'Update' button and reopening the job parameters, all of the parameters disappear. Could someone advise on how to do fix this issue or an alternate way to pass parameters to several glue jobs and trigger crawlers on a schedule? They are custom parameters that I am adding to parse through via the argument parser. For example, I add a parameter to identify the primary key as an input for a job as well as read and write paths.

  • secondabhi_aws EXPERT
    9 months ago

    What are the parameters, can you please mention those here.

Topics
AnalyticsDatabase
Tags
AWS GlueExtract Transform & Load Data
Language
English
Nick
asked 9 months ago254 views
1 Answer
0

Hello,

I understand that you are trying to save job parameters to pass to individual jobs within the workflow. But after adding them and reopening the job parameters, they do not appear on the console.

To rule out any permission issues, could you please verify if you have the below required permissions to update the workflow run properties.

UpdateWorkflow action - Updates an existing workflow.
PutWorkflowRunProperties action - Puts the specified workflow run properties for the given workflow run.
GetWorkflowRunProperties action - Retrieves the workflow run properties which were set during the run.

You can also check your cloudtaril logs to see if there are any errors while saving the parameters.

Please refer below documentation for more details:
[+] https://docs.aws.amazon.com/glue/latest/dg/workflow-run-properties-code.html
[+] https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-workflow.html#aws-glue-api-workflow-PutWorkflowRunProperties
[+] https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-workflow.html#aws-glue-api-workflow-UpdateWorkflow

Also, please try clearing your browser cache and try again and use a different browser once to see if that helps.

Additionally, you can use the get-workflow-run-properties(https://docs.aws.amazon.com/cli/latest/reference/glue/get-workflow-run-properties.html) CLI command to check if the run properties were set for the workflow or not.

AWS
Ankur_J
answered 9 months ago
  • Nick
    9 months ago

    I have admin privileges and this issue is persisting through several browsers over a multi-day period. I receive a green bar at the top that indicates my parameters were saved but upon opening and refreshing, the parameters still disappear. I reference these parameters to create other workflow jobs as well, so the GUI is helpful in that respect.

  • Nick
    9 months ago

    Additionally, some of my jobs are now failing because they parameters are not being passed to the jobs

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content