What is the most cost efficient and fastest way to start GLUE ETL development
We are going to start development of GLUE ETLs . The different options available are :
- From the AWS Console : This seems to be costly and slow and not very efficient for developing scripts
- From Dev Endpoints : Billing Rate is high
- By AWS Glue Docker Image : Lacks functionality
- Interactive Sessions
- Local Setup
Which among these will be simple to set up and not incur cost ?
I would say the most cost efficient,simple and fastest way to start Glue ETL development are:
- Use Glue Docker image
- Use Interactive session
It always depend on the actual priorities you have to define the trade off between cost efficiency and developer experience.
The most cost efficient I would agree is using the docker container . If you prefer to use a traditional IDE environment instead of a notebook you should be able to use it for that as well. Performance will depend on the local machine.
If you give more weight to a flexible developer experience using notebooks and performance when running the jobs or cells then Interactive sessions would be a better choice. You can install your notebook with all extensions and tune the number of DPUs you want to use changing that between sessions.
If the developer is fine with a managed notebook, Glue Studio Notebooks would have the same cost of Interactive sessions (it just depends on the DPU you select for the session and duration of the session), same configuration based on session . Glue Studio Notebooks are only available in some regions currently.
hope this helps
Is there a way to increase DPU or Spark config on the Docker ?
Yes , I agree, I am more inclined to use the docker image , but I faces quite a lot of challenges and still facing , in setting it up. For starter , the Jupyter notebook does not have nbextensions preinstalled ,and I could not get it installed either , asked a separate question on that . Then the SPARK is very slow .
Havent tried the interative sessions , will start with it soon .
what are advantages of running ETL jobs in aws glue?asked 23 days ago
What is the best practice to load data to redshift with aws glue ?asked 2 years ago
Glue ETL PySpark Job Fails after Upgrade from Glue Version 2.0 to 3.0 error occurred while calling pyWriteDynamicFrame EOFException occurred while reading the port number from pyspark.daemon's stdoutasked 5 months ago
Development Endpoint & Glue Version 1.0asked 3 months ago
ETL automation and testingasked 4 months ago
AWS Glue 2.0/3.0 are not available in Dev endpointsAccepted Answerasked 7 months ago
What is the most cost efficient and fastest way to start GLUE ETL developmentAccepted Answerasked a month ago
Cost of Athena and glueasked 7 months ago
Need inputs on Glue to move data from Aurora MySQL to Redshiftasked 25 days ago
What is the right setup for development for a team who works on the same project?Accepted Answerasked 3 months ago