How to reference a glue job python argument in CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

0

I would like to reference a python parameter of a glue job within the CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

For instance: I have an argument --s3_location which has to be referred within S3CsvSource Node of my ETL job as given below

s3_source_node1 = glueContext.create_dynamic_frame.from_options(
    format_options={
        "quoteChar": '"',
        "withHeader": True,
        "separator": ",",
        "optimizePerformance": False,
    },
    connection_type="s3",
    format="csv",
    connection_options={"paths": [args["s3_location"]]},
    transformation_ctx="s3_source_node1",
)

This has to be done via create_job() function available as part of GlueClient in boto3. However while defining CodeGenConfigurationNodes within create_job(), I was not able to mention [args["s3_location"] in path property of S3CsvSource node property.

Current CodeGenConfigurationNodes for S3CsvSource is as below:

CodeGenConfigurationNodes = {
    'node-1':{
        'S3CsvSource': {
                'Name': 's3_source',
                'Paths': [
                    's3://my_bucket/sample_input.csv',
                ],
                'Separator': 'comma',
                'QuoteChar': 'quote',
                'WithHeader': True,
                'WriteHeader': True,
       },
}]

My expected result is something like

CodeGenConfigurationNodes = {
    'node-1':{
        'S3CsvSource': {
                'Name': 's3_source',
                'Paths': [
                    args["s3_location"],
                ],
                'Separator': 'comma',
                'QuoteChar': 'quote',
                'WithHeader': True,
                'WriteHeader': True,
       },
}]

Where args["s3_location"] refers to glue job parameter --s3_location.

  • Does it throw any error when you reference args["s3_location"]?

asked 3 months ago28 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions