How to reference a glue job python argument in CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

0

I would like to reference a python parameter of a glue job within the CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

For instance: I have an argument --s3_location which has to be referred within S3CsvSource Node of my ETL job as given below

s3_source_node1 = glueContext.create_dynamic_frame.from_options(
    format_options={
        "quoteChar": '"',
        "withHeader": True,
        "separator": ",",
        "optimizePerformance": False,
    },
    connection_type="s3",
    format="csv",
    connection_options={"paths": [args["s3_location"]]},
    transformation_ctx="s3_source_node1",
)

This has to be done via create_job() function available as part of GlueClient in boto3. However while defining CodeGenConfigurationNodes within create_job(), I was not able to mention [args["s3_location"] in path property of S3CsvSource node property.

Current CodeGenConfigurationNodes for S3CsvSource is as below:

CodeGenConfigurationNodes = {
    'node-1':{
        'S3CsvSource': {
                'Name': 's3_source',
                'Paths': [
                    's3://my_bucket/sample_input.csv',
                ],
                'Separator': 'comma',
                'QuoteChar': 'quote',
                'WithHeader': True,
                'WriteHeader': True,
       },
}]

My expected result is something like

CodeGenConfigurationNodes = {
    'node-1':{
        'S3CsvSource': {
                'Name': 's3_source',
                'Paths': [
                    args["s3_location"],
                ],
                'Separator': 'comma',
                'QuoteChar': 'quote',
                'WithHeader': True,
                'WriteHeader': True,
       },
}]

Where args["s3_location"] refers to glue job parameter --s3_location.

  • Does it throw any error when you reference args["s3_location"]?

gefragt vor 2 Jahren109 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen