from pyspark.sql.functions import ArrayType
For this I am getting error in Glue job.
Glue Version I am using is 3.0 which supports spark 3.1
ImportError: cannot import name 'ArrayType' from 'pyspark.sql.functions' (/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/functions.py)
On using next Glue version Glue 4.0 with spark 3.3 getting another error
from pyspark.sql.functions import ArrayType
from pyspark.sql.functions import StringType
class JobBase(object):
#all udf has to be initialised as first step
def __init__(self):
print("Inside the constructor of Class phases ")
self.winner_org_calculation_udf=udf(JobBase.winner_org_calculation, ArrayType(StringType()))
TypeError: ArrayType.__init__() missing 1 required positional argument: 'elementType'