By using AWS re:Post, you agree to the Terms of Use
/What is the most cost efficient and fastest way to start GLUE ETL development/

What is the most cost efficient and fastest way to start GLUE ETL development

0

We are going to start development of GLUE ETLs . The different options available are :

  1. From the AWS Console : This seems to be costly and slow and not very efficient for developing scripts
  2. From Dev Endpoints : Billing Rate is high
  3. By AWS Glue Docker Image : Lacks functionality
  4. Interactive Sessions
  5. Local Setup

Which among these will be simple to set up and not incur cost ?

asked a month ago28 views
3 Answers
0
Accepted Answer

Hello,

I would say the most cost efficient,simple and fastest way to start Glue ETL development are:

  1. Use Glue Docker image
  2. Use Interactive session

References:

[1] https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container/

[2] https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions.html

answered a month ago
EXPERT
reviewed a month ago
0

Hi,

It always depend on the actual priorities you have to define the trade off between cost efficiency and developer experience.

The most cost efficient I would agree is using the docker container . If you prefer to use a traditional IDE environment instead of a notebook you should be able to use it for that as well. Performance will depend on the local machine.

If you give more weight to a flexible developer experience using notebooks and performance when running the jobs or cells then Interactive sessions would be a better choice. You can install your notebook with all extensions and tune the number of DPUs you want to use changing that between sessions.

If the developer is fine with a managed notebook, Glue Studio Notebooks would have the same cost of Interactive sessions (it just depends on the DPU you select for the session and duration of the session), same configuration based on session . Glue Studio Notebooks are only available in some regions currently.

hope this helps

EXPERT
answered a month ago
  • Is there a way to increase DPU or Spark config on the Docker ?

0

Yes , I agree, I am more inclined to use the docker image , but I faces quite a lot of challenges and still facing , in setting it up. For starter , the Jupyter notebook does not have nbextensions preinstalled ,and I could not get it installed either , asked a separate question on that . Then the SPARK is very slow .

Havent tried the interative sessions , will start with it soon .

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions