EMR automatically finishing when manually finishing is selected via Airflow

0

So I define manually finishing using the RunJobFlow operator (https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) "KeepJobFlowAliveWhenNoSteps": True. However, the cluster cluster automatically finishes but says terminated by user request.

EMR config

Terminated by user request, which is not true

This is the configuration I have defined in airflow:

JOB_FLOW_OVERRIDES = {
    "Name": "X",
    "Applications": [
        {"Name": "Spark"},
        {"Name": "Ganglia"},
        ],
    "Instances": {
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        "Ec2SubnetId":"X",
        "EmrManagedMasterSecurityGroup": "X",
        "EmrManagedSlaveSecurityGroup": "X",
        "ServiceAccessSecurityGroup": "X",
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "SPOT",
                "InstanceRole": "MASTER",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 1,
            },
            {
                "Name": "Core nodes",
                "Market": "SPOT",
                "InstanceRole": "CORE",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 2,
            }
        ],
        "Ec2KeyName": "X",
    },
    "LogUri": "X",
    "ReleaseLabel": "X",
    "VisibleToAllUsers": True,
    "JobFlowRole": "DataPipelineDefaultResourceRole",
    "ServiceRole": "DataPipelineDefaultRole",
}
已提问 5 个月前151 查看次数
1 回答
3

Hello,

Please check the CloudTrail and see which user initiates the TerminateJobFlows API call. There could be the user or the Org policy who gain access to terminate the cluster would be performed this. For an instance, if any policy that created in your Org when the cluster created using public subnet will be terminated automatically. You can get more insights from referring the CloudTrail. If the TerminateJobFlows triggered by Airflow role, then it is handled as part of DAG execution.

AWS
支持工程师
已回答 5 个月前
profile picture
专家
已审核 5 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则