AWS announces AWS Interconnect – multicloud (preview), providing simple, resilient, high-speed private connections to other cloud service providers. AWS Interconnect - multicloud is easy to configure and provides high-speed, resilient connectivity with dedicated bandwidth, enabling customers to interconnect AWS networking services such as AWS Transit Gateway, AWS Cloud WAN, and Amazon VPC to other cloud service providers with ease.
Glue Studio Designer: use pyspark functions
I designed a glue job using Glue studio designer canvas feature and am using a custom transform in there. However, I am struggling to leverage functions like the ones from 'from pyspark.sql.functions import *' in the code as I get error "NameError: name xyz not defined.
How can I leverage these functions in the canvas tool?
- Newest
- Most votes
- Most comments
Try importing specific function instead of import *. For example, "from pyspark.sql.functions import split" to import split function.
I tried replicating your problem, it complained that import * can only be used at module level. But when I changed to specific function, it worked.
Hope this help.
Hi ,
yes, any library you need for your custom transform should be imported within the function.
just to consider if you want to run SparkSQL you could also use the SQL transform.
hope this helps,
Relevant content
- asked 3 years ago
- AWS OFFICIALUpdated 10 months ago
so is it a best practice to run the imports within the function fo the custom transform?