- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
To Add to the question above , I would recommend that you move to Glue Interactive Sessions.
Dev end-points are no more developed and do support only Glue version 1.
Switching to Interactive sessions, you can dynamically choose the Glue version (2,3 or 4) and hence spark version. Furthermore it highly simplify the way you can add additional python modules by using a magics
According to my understanding, you are including the custom module by uploading to the local directory structure of your Glue Dev endpoint using SSH and trying to link them. However, according to the documentation, the way you import custom python modules that you use for your development endpoint is by adding them as a dependency S3 path at the time of creation of the development endpoint.
Python library path
Comma-separated Amazon Simple Storage Service (Amazon S3) paths to Python libraries that are required by your script. Multiple values must be complete paths separated by a comma (,). Only individual files are supported, not a directory path.
In order to use your custom libraries, generate a .whl
file of your custom module and upload to an S3 path and use it as a parameter with "Python library path" in dev endpoint.
For more information on how you can create a .whl
file refer Python Official documentation or use the following pip command in the directory of the python package with the latest pip and wheel:
pip wheel .
If this still doesn't resolve the issue, I would recommend trying out different options like Glue Interactive Sessions/Notebooks
Unlike AWS Glue development endpoints, AWS Glue interactive sessions are serverless with no infrastructure to manage. You can start interactive sessions very quickly. Interactive sessions have a 1-minute billing minimum with cost-control features. This reduces the cost of developing data preparation applications.
Contenuto pertinente
- Come posso risolvere gli errori di connessione di Marketplace AWS nei miei processi ETL in AWS Glue?AWS UFFICIALEAggiornata 3 mesi fa
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata un anno fa