By using AWS re:Post, you agree to the Terms of Use

AWS Glue 2.0/3.0 are not available in Dev endpoints

0

I am trying to create a Dev endpoint in AWS Glue 2.0 or higher and use a Notebook to attach to this env. However, the only 3 options I see are for Glue <= 1.0. Please note, the higher versions are indeed available in Glue Jobs. Any idea why this is the case and how do I resolve this ?

asked 10 months ago108 views
2 Answers
1
Accepted Answer

Dev Endpoints only support Glue 1.0, they will not be upgraded to support Glue 2.0 or later.

For interactive development with Glue 2.0 or later , please review and test either Glue studio notebooks or Glue Interactive Sessions.

Please, note these feature are in public preview (available in selected regions us-east-1,us-west-2,eu-west-1,ap-northeast-1) , so the full feature set will be available at General Availability , and there still might be some rough edges.

For more information on these new feature look at this documentation section.

Specific info on Interactive Sessions are here; while details on Glue studio notebooks are here.

EXPERT
answered 10 months ago
  • Thanks much for the response! Can you please let me know what is 'GA' & how do I select either of the two mechanisms ? I don't see the Glue Studio Notebook available to be used, I suppose it needs to be activated somehow. Any pointers regarding this is very much appreciated.

  • @AWS-User-1825611 see the edited answer, preview is limited to few regions.

  • Thanks much for your help!

  • Hello. For clarification, it sounds like you're saying that AWS will never update the Glue version supported for Glue development endpoints. When Glue 1.0 gets deprecated does that mean that Glue development endpoint will go away as well? We should switch to Glue Studio for interactive development going forward?

0

From the docs:

Note Development endpoints are not supported for use with AWS Glue version 2.0 jobs. For more information, see Running Spark ETL Jobs with Reduced Startup Times.

answered 10 months ago
  • Please, add a link to the documentation section you are referring to enhance the user experience.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions