How do you pass special parameters for AWS Glue jobs via AWS CloudFormation

Question

When creating a AWS Glue ETL Job with AWS CloudFormation, how do you specify advanced options such as additional JAR's that the job may require, special security configuration parameters for KMS encryption, etc?

Accepted Answer

Hi,

In AWS CloudFormation when using the Resource AWS::Glue::Job under Properties you need to add these advanced parameter in the  [DefaultArguments][1] section.

"DefaultArguments": {
                "--TempDir": "s3://aws-glue-temporary-633748029103-eu-west-1/serhat",
                "--class": "GlueApp",
                "--enable-glue-datacatalog": "",
                "--enable-metrics": "",
                "--encryption-type": "sse-s3",
                "--job-bookmark-option": "job-bookmark-enable",
                "--job-language": "scala"
            },

You can find the list of parameters [here][2].

For additional JARS: **--extra-jars** is the parameter.

For Server Side Encryption: **--encryption-type** is the parameter.

Hope this helps.

[1]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-glue-job.html#cfn-glue-job-defaultarguments
  [2]: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html#aws-glue-api-crawler-pyspark-extensions-python-intro-parameters

How do you pass special parameters for AWS Glue jobs via AWS CloudFormation

관련 콘텐츠