AWS Glue Redshift annotation

Question

Hello everyone.

I am trying to generate a classic script through Glue Studio to get the annotations of my source, transformations and sink in one of my jobs, so far I have been able to get without problem the annotations of my source and transformations but I can't get the annotations of the sink.
My sink is simply to write data in a Redshift table, then I am putting in Glue Studio "Data Target - Redshift" but at the moment of generating the classic script I get the following error:
"Error generating the script
[gluestudio-service.[MASKED].amazonaws.com] createScript: InvalidInputException: Invalid DataSink: DataSink(name=Amazon Redshift, classification=DataSink, type=Redshift, inputs=[node-2], isSinkInStreamingDAG=false)".

Has anyone else had this happen to you?.

Or do you have the correct annotation for Redshift sink?.

Answer

Hello,

I have tested by generating a sample script by just selecting S3 as input/source and Redshift as target/sink and I do not see annotation generated in the script for Redshift where as a regular Glue ETL job generated annotation as below:

=================
## @type: DataSink
## @args: [catalog_connection = "redshiftconnection", connection_options = {"dbtable": "testtable", "database": "sample"}, redshift_tmp_dir = TempDir, transformation_ctx = "datasink4"]
## @return: datasink4
## @inputs: [frame = dropnullfields3]
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = dropnullfields3, catalog_connection = "redshiftconnection", connection_options = {"dbtable": "testtable", "database": "sample"}, redshift_tmp_dir = args["TempDir"], transformation_ctx = "datasink4")

=================

Confirmed with Glue expert that this is normal behavior with Glue Studio and please raise a support ticket with us with your use case and if it's a blocker, we will raise a PFR for it with our Glue service team.

AWS Glue Redshift annotation

@type: DataSink

@args: [catalog_connection = "redshiftconnection", connection_options = {"dbtable": "testtable", "database": "sample"}, redshift_tmp_dir = TempDir, transformation_ctx = "datasink4"]

@return: datasink4

@inputs: [frame = dropnullfields3]

Relevanter Inhalt