I would like to reference a python parameter of a glue job within the CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.
For instance:
I have an argument --s3_location
which has to be referred within S3CsvSource Node of my ETL job as given below
s3_source_node1 = glueContext.create_dynamic_frame.from_options(
format_options={
"quoteChar": '"',
"withHeader": True,
"separator": ",",
"optimizePerformance": False,
},
connection_type="s3",
format="csv",
connection_options={"paths": [args["s3_location"]]},
transformation_ctx="s3_source_node1",
)
This has to be done via create_job()
function available as part of GlueClient in boto3.
However while defining CodeGenConfigurationNodes
within create_job()
, I was not able to mention [args["s3_location"]
in path
property of S3CsvSource
node property.
Current CodeGenConfigurationNodes for S3CsvSource is as below:
CodeGenConfigurationNodes = {
'node-1':{
'S3CsvSource': {
'Name': 's3_source',
'Paths': [
's3://my_bucket/sample_input.csv',
],
'Separator': 'comma',
'QuoteChar': 'quote',
'WithHeader': True,
'WriteHeader': True,
},
}]
My expected result is something like
CodeGenConfigurationNodes = {
'node-1':{
'S3CsvSource': {
'Name': 's3_source',
'Paths': [
args["s3_location"],
],
'Separator': 'comma',
'QuoteChar': 'quote',
'WithHeader': True,
'WriteHeader': True,
},
}]
Where args["s3_location"]
refers to glue job parameter --s3_location
.
Does it throw any error when you reference args["s3_location"]?