AWS Glue Job - Extract the Job related metadata

0

Hi,

we have 50 Glue jobs deployed in our AWS account . I want to be able to extract the job related configuraion metadta as a csv files. The metadata that I want to extract are

  1. script file path
  2. Script File location 3)Job Type
  3. Jar files list
  4. Referenced file Path
  5. Job Language

Is there a command line option or a python script that I can generate to extract this information easily ?

SMR
已提问 10 个月前510 查看次数
2 回答
3
已接受的回答

I understand that you may be finding little difficulty in filtering the results, specially for default arguments as that contains special characters("--") and to filter those parameters, syntax would be little different. Usual syntax won't work for default arguments.

Here is how the CLI command would look like for your use case:

aws glue get-jobs --query 'Jobs[].[Name,'Command.Name','Command.ScriptLocation','GlueVersion',DefaultArguments."--extra-py-files",DefaultArguments."--job-language",DefaultArguments."--TempDir",DefaultArguments."--extra-files"]' --profile <profile_name> --output table

References:

Glue CLI

CLI Filter Usage

Hope you find this useful.

Abhishek

profile pictureAWS
专家
已回答 10 个月前
profile pictureAWS
专家
iBehr
已审核 10 个月前
0

Hi,

There is a CLI command for this. It is get-jobs: see https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/get-jobs.html

You can then use option: --output table to get a tabular format.

Also, you may want to filter only some of the returned fields (they can be abundant with command get-jobs ....) by using --query option: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

Best,

Didier

profile pictureAWS
专家
已回答 10 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则