How to update the value in AWS Glue Job parameter via CLI?

Question

Hi Everyone,

I want to change the Job parameters in AWS Glue each time based on my requirement, I need to automate this. I do have a text file which consists of the parameters list. Is there anyway to automate this process via CLI or any custom script to process the parameters one by one.

Thanks,
Krishna

Answer

Hello Krishna,

It's totally possible to pass parameter to a glue job in an automated way. You can do it in two ways: updating the job definition or passing parameters when starting a job run, which one you will use depends on your solution.

You can accomplish this by using the available aws sdk for your programming language of choice or by invoking the API through CLI. I will leave here the documentation for the CLI and the boto3 (aws sdk for python) module, for both the **update job** and **start job run** calls:

[bot3.Glue.Client.update_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/update_job.html)

[boto3.Glue.Client.start_job_run](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/start_job_run.html)

[aws glue update-job](https://docs.aws.amazon.com/cli/latest/reference/glue/update-job.html)

[aws glue start-job-run](https://docs.aws.amazon.com/cli/latest/reference/glue/start-job-run.html)

The AWS SDK is avalailable on other programming languages too, check here for all available languages: https://aws.amazon.com/developer/tools/

It seems to me, based on your question, that what you want is the start job run, to start the job each time with a different parameter from a lists of values.

Hope this helps you. Let me know if you have any further questions!

Have a nice day.

How to update the value in AWS Glue Job parameter via CLI?

Relevant content