Aws glue script toDF().sort() method gives exception

Hello,

I would like to inform above exception generally occurs when spark is not able to find conditional columns in dataset.

To confirm , I have tested sort and orderBy function in Glue job and it is working absolutely fine. Please find the sample code below:

++++++++++ datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "testdb", table_name = "nycflights13_csv", transformation_ctx = "datasource0")

datasource0.toDF().sort('year','month').show(5)

datasource0.toDF(). orderBy('year','month').show(5) ++++++++++

I would request you please verify schema once again and try to print sample data after creating the dynamic frame and then use sort or orderBy function:

+++++++++ DyanmicFrame.printSchema()

##Above function should print the columns which you would like to use in sort or orderBY

DyanmicFrame.toDF().show()

##Above function should return values

DyanmicFrame.toDF().sort('year','month').show(5)

DyanmicFrame.toDF(). orderBy('year','month').show(5) +++++++++++

If you still face any issue, Please feel free to reach out to AWS Premium Support with sample data and we will be happy to help.

Have a Nice day!