By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Glue Jupyter Job , Execution Class

0

How can one set up an Execution Class = FLEX on a Jupyter Job Run , im using the %magic on my %%configure cell like below and also setting the input arguments with --execution_class = FLEX But still the jobs are quicking as STANDARD

%%configure { "region": "us-east-1", "idle_timeout": "480", "glue_version": "3.0", "number_of_workers": 10, "execution_class": "FLEX", "worker_type": "G.1X" }

Enter image description here

Topics
AnalyticsDatabase
Tags
AnalyticsAWS GlueExtract Transform & Load Data
Language
English
Jorge Vidinha
asked a year ago547 views
2 Answers
1

Sorry,me too...

Although it is a rather peculiar means, I was able to change the execution class to FLEX as follows.

  1. Push Glue to CodeCommit.
  2. On CodeCommit, change the value of "executionClass" to "FLEX" in [Job name].json Enter image description here
  3. Pull from CodeCommit.

With the above operation, the setting of FLEX has been changed. Enter image description here Enter image description here

profile picture
ARBR
answered a year ago
  • Jorge Vidinha
    a year ago

    Thanks ARBR , Im not using Codecommit at this point so really searching for a solution from console jupyter job runs

  • ARBR
    a year ago

    By comparing before and after, we found the following:

    Under "jobConfig", the following two settings were added.

    • "flexExecution": true
    • "minFlexWorkers": null

    Adding these to the file downloaded with [Action]-[Download] and uploading with [Action]-[Upload] enabled "Flex".

    However, the statement below did not work.

    %%configure
{
    "flex_execution" : "true",
    "min_flex_workers" : "null"
}
1

I haven't tried it yet, but the URL below seems to be helpful.

https://aws.amazon.com/jp/blogs/big-data/introducing-aws-glue-flex-jobs-cost-savings-on-etl-workloads/

According to the information on the site, the name of the parameter seems to be "execution-class" (instead of "execution_class").

profile picture
ARBR
answered a year ago
  • Jorge Vidinha
    a year ago

    Thanks for the help ...

    On interactive Jupyter Jobs I have set up the the execution class to FLEX tried the 2 variants below: A "execution_class": "FLEX" B "execution-class": "FLEX" And also added execution-class = FLEX to JOB Run with job parameters at no success, it still shows Execution Class Standard .

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content