使用AWS re:Post即您表示您同意 AWS re:Post 使用条款
AWS re:Postre:Post

Glue Jupyter Job , Execution Class

0

How can one set up an Execution Class = FLEX on a Jupyter Job Run , im using the %magic on my %%configure cell like below and also setting the input arguments with --execution_class = FLEX But still the jobs are quicking as STANDARD

%%configure { "region": "us-east-1", "idle_timeout": "480", "glue_version": "3.0", "number_of_workers": 10, "execution_class": "FLEX", "worker_type": "G.1X" }

Enter image description here

主题
分析数据库
标签
分析AWS Glue提取变换和加载数据
语言
English
Jorge Vidinha
已提问 1 年前560 查看次数
2 回答
1

Sorry,me too...

Although it is a rather peculiar means, I was able to change the execution class to FLEX as follows.

  1. Push Glue to CodeCommit.
  2. On CodeCommit, change the value of "executionClass" to "FLEX" in [Job name].json Enter image description here
  3. Pull from CodeCommit.

With the above operation, the setting of FLEX has been changed. Enter image description here Enter image description here

profile picture
ARBR
已回答 1 年前
  • Jorge Vidinha
    1 年前

    Thanks ARBR , Im not using Codecommit at this point so really searching for a solution from console jupyter job runs

  • ARBR
    1 年前

    By comparing before and after, we found the following:

    Under "jobConfig", the following two settings were added.

    • "flexExecution": true
    • "minFlexWorkers": null

    Adding these to the file downloaded with [Action]-[Download] and uploading with [Action]-[Upload] enabled "Flex".

    However, the statement below did not work.

    %%configure
{
    "flex_execution" : "true",
    "min_flex_workers" : "null"
}
1

I haven't tried it yet, but the URL below seems to be helpful.

https://aws.amazon.com/jp/blogs/big-data/introducing-aws-glue-flex-jobs-cost-savings-on-etl-workloads/

According to the information on the site, the name of the parameter seems to be "execution-class" (instead of "execution_class").

profile picture
ARBR
已回答 1 年前
  • Jorge Vidinha
    1 年前

    Thanks for the help ...

    On interactive Jupyter Jobs I have set up the the execution class to FLEX tried the 2 variants below: A "execution_class": "FLEX" B "execution-class": "FLEX" And also added execution-class = FLEX to JOB Run with job parameters at no success, it still shows Execution Class Standard .

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈，并能促进提问者的职业发展。

回答问题的准则

相关内容