2 réponses
- Le plus récent
- Le plus de votes
- La plupart des commentaires
1
Try importing specific function instead of import *. For example, "from pyspark.sql.functions import split" to import split function.
I tried replicating your problem, it complained that import * can only be used at module level. But when I changed to specific function, it worked.
Hope this help.
répondu il y a 2 ans
0
Hi ,
yes, any library you need for your custom transform should be imported within the function.
just to consider if you want to run SparkSQL you could also use the SQL transform.
hope this helps,
Contenus pertinents
- demandé il y a 7 mois
- demandé il y a un mois
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a 3 ans
- AWS OFFICIELA mis à jour il y a 3 ans
- AWS OFFICIELA mis à jour il y a 3 ans
- AWS OFFICIELA mis à jour il y a 3 ans
so is it a best practice to run the imports within the function fo the custom transform?