Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Glue Jupyter Job , Execution Class

0

How can one set up an Execution Class = FLEX on a Jupyter Job Run , im using the %magic on my %%configure cell like below and also setting the input arguments with --execution_class = FLEX But still the jobs are quicking as STANDARD

%%configure { "region": "us-east-1", "idle_timeout": "480", "glue_version": "3.0", "number_of_workers": 10, "execution_class": "FLEX", "worker_type": "G.1X" }

Enter image description here

Topics
AnalyticsDatabase
Tags
AnalyticsExtract Transform & Load DataAWS Glue
Language
English
asked 3 years ago897 views
2 Answers
1

Sorry,me too...

Although it is a rather peculiar means, I was able to change the execution class to FLEX as follows.

  1. Push Glue to CodeCommit.
  2. On CodeCommit, change the value of "executionClass" to "FLEX" in [Job name].json Enter image description here
  3. Pull from CodeCommit.

With the above operation, the setting of FLEX has been changed. Enter image description here Enter image description here

answered 3 years ago
  • Jorge Vidinha
    3 years ago

    Thanks ARBR , Im not using Codecommit at this point so really searching for a solution from console jupyter job runs

  • ARBR
    3 years ago

    By comparing before and after, we found the following:

    Under "jobConfig", the following two settings were added.

    • "flexExecution": true
    • "minFlexWorkers": null

    Adding these to the file downloaded with [Action]-[Download] and uploading with [Action]-[Upload] enabled "Flex".

    However, the statement below did not work.

    %%configure
{
    "flex_execution" : "true",
    "min_flex_workers" : "null"
}
1

I haven't tried it yet, but the URL below seems to be helpful.

https://aws.amazon.com/jp/blogs/big-data/introducing-aws-glue-flex-jobs-cost-savings-on-etl-workloads/

According to the information on the site, the name of the parameter seems to be "execution-class" (instead of "execution_class").

answered 3 years ago
  • Jorge Vidinha
    3 years ago

    Thanks for the help ...

    On interactive Jupyter Jobs I have set up the the execution class to FLEX tried the 2 variants below: A "execution_class": "FLEX" B "execution-class": "FLEX" And also added execution-class = FLEX to JOB Run with job parameters at no success, it still shows Execution Class Standard .

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content