AWS Glue Job - Extract the Job related metadata

Question

Hi,

we have 50 Glue jobs deployed in our AWS account . I want to be able to extract the job related configuraion metadta as a csv files. The metadata that I want to extract are 
1) script file path
2) Script File location
3)Job Type
4) Jar files list
5) Referenced file Path
6) Job Language

Is there a command line option or a python script that I can generate to extract this information easily ?

Accepted Answer

I understand that you may be finding little difficulty in filtering the results, specially for default arguments as that contains special characters("--") and to filter those parameters, syntax would be little different. Usual syntax won't work for default arguments.

Here is how the CLI command would look like for your use case:

aws glue get-jobs --query 'Jobs[].[Name,'Command.Name','Command.ScriptLocation','GlueVersion',DefaultArguments."--extra-py-files",DefaultArguments."--job-language",DefaultArguments."--TempDir",DefaultArguments."--extra-files"]' --profile  --output table

References:

[Glue CLI](https://docs.aws.amazon.com/cli/latest/reference/glue/index.html)

[CLI Filter Usage](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html)

Hope you find this useful.

Abhishek

Answer

Hi,

There is a CLI command for this. It is get-jobs: see https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/get-jobs.html

You can then use option: --output table to get a tabular format.

Also, you may want to filter only some of the returned fields (they can be abundant with command get-jobs ....)  by using --query option: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

Best,

Didier

AWS Glue Job - Extract the Job related metadata

Relevant content