- Newest
- Most votes
- Most comments
Hello,
I would like to inform above exception generally occurs when spark is not able to find conditional columns in dataset.
To confirm , I have tested sort and orderBy function in Glue job and it is working absolutely fine. Please find the sample code below:
++++++++++ datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "testdb", table_name = "nycflights13_csv", transformation_ctx = "datasource0")
datasource0.toDF().sort('year','month').show(5)
datasource0.toDF(). orderBy('year','month').show(5) ++++++++++
I would request you please verify schema once again and try to print sample data after creating the dynamic frame and then use sort or orderBy function:
+++++++++ DyanmicFrame.printSchema()
##Above function should print the columns which you would like to use in sort or orderBY
DyanmicFrame.toDF().show()
##Above function should return values
DyanmicFrame.toDF().sort('year','month').show(5)
DyanmicFrame.toDF(). orderBy('year','month').show(5) +++++++++++
If you still face any issue, Please feel free to reach out to AWS Premium Support with sample data and we will be happy to help.
Have a Nice day!
Relevant content
- asked a year ago
- asked a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 3 years ago
Hi @Shubham_P, is there a way to sort() or orderBy() a Dynamic Dataframe avoiding going .toDF() ?
Thanks