How to send SNS notification from a Glue Streaming job and include the contents of a dataframe in the message body?

Question

0
Hello, I am running a glue streaming job where I read data from KDS, do some aggregation and filter it. I want to trigger SNS on this filtered DF and include its column values in the SNS message body. If I write data directly to S3 from the filtered DF without adding any code for calling SNS, it works fine. If I add the logic for calling SNS on the filtered DF, SNS gets triggered but somehow the filtered df is sent empty and nothing gets written to S3. What is the best way to do it? Many thanks.

Gonzalo Herreros · Answer

Inside the forEach function, get the DataFrame contents with collect() and then use it to call the SNS client using boto3.   
Make sure the DataFrame contents are small so it cannot break, for instance using take() instead of collect if you know that makes the volume of data retrieved predictable.

How to send SNS notification from a Glue Streaming job and include the contents of a dataframe in the message body?

Relevant content