By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Extract Transform & Load Data

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to reference a glue job python argument in CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

I would like to reference a python parameter of a glue job within the CodeGenConfigurationNodes of create_job() function in boto3 Glue Client. For instance: I have an argument `--s3_location` which has to be referred within S3CsvSource Node of my ETL job as given below ``` s3_source_node1 = glueContext.create_dynamic_frame.from_options( format_options={ "quoteChar": '"', "withHeader": True, "separator": ",", "optimizePerformance": False, }, connection_type="s3", format="csv", connection_options={"paths": [args["s3_location"]]}, transformation_ctx="s3_source_node1", ) ``` This has to be done via `create_job()` function available as part of GlueClient in boto3. However while defining `CodeGenConfigurationNodes` within `create_job()`, I was not able to mention `[args["s3_location"]` in `path` property of `S3CsvSource` node property. Current CodeGenConfigurationNodes for S3CsvSource is as below: ``` CodeGenConfigurationNodes = { 'node-1':{ 'S3CsvSource': { 'Name': 's3_source', 'Paths': [ 's3://my_bucket/sample_input.csv', ], 'Separator': 'comma', 'QuoteChar': 'quote', 'WithHeader': True, 'WriteHeader': True, }, }] ``` My expected result is something like ``` CodeGenConfigurationNodes = { 'node-1':{ 'S3CsvSource': { 'Name': 's3_source', 'Paths': [ args["s3_location"], ], 'Separator': 'comma', 'QuoteChar': 'quote', 'WithHeader': True, 'WriteHeader': True, }, }] ``` Where `args["s3_location"]` refers to glue job parameter `--s3_location`.
0
answers
0
votes
27
views
asked a month ago