By using AWS re:Post, you agree to the Terms of Use

Unanswered Questions tagged with AWS Glue

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to reference a glue job python argument in CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

I would like to reference a python parameter of a glue job within the CodeGenConfigurationNodes of create_job() function in boto3 Glue Client. For instance: I have an argument `--s3_location` which has to be referred within S3CsvSource Node of my ETL job as given below ``` s3_source_node1 = glueContext.create_dynamic_frame.from_options( format_options={ "quoteChar": '"', "withHeader": True, "separator": ",", "optimizePerformance": False, }, connection_type="s3", format="csv", connection_options={"paths": [args["s3_location"]]}, transformation_ctx="s3_source_node1", ) ``` This has to be done via `create_job()` function available as part of GlueClient in boto3. However while defining `CodeGenConfigurationNodes` within `create_job()`, I was not able to mention `[args["s3_location"]` in `path` property of `S3CsvSource` node property. Current CodeGenConfigurationNodes for S3CsvSource is as below: ``` CodeGenConfigurationNodes = { 'node-1':{ 'S3CsvSource': { 'Name': 's3_source', 'Paths': [ 's3://my_bucket/sample_input.csv', ], 'Separator': 'comma', 'QuoteChar': 'quote', 'WithHeader': True, 'WriteHeader': True, }, }] ``` My expected result is something like ``` CodeGenConfigurationNodes = { 'node-1':{ 'S3CsvSource': { 'Name': 's3_source', 'Paths': [ args["s3_location"], ], 'Separator': 'comma', 'QuoteChar': 'quote', 'WithHeader': True, 'WriteHeader': True, }, }] ``` Where `args["s3_location"]` refers to glue job parameter `--s3_location`.
0
answers
0
votes
23
views
asked 17 days ago