EMR automatically finishing when manually finishing is selected via Airflow

0

So I define manually finishing using the RunJobFlow operator (https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) "KeepJobFlowAliveWhenNoSteps": True. However, the cluster cluster automatically finishes but says terminated by user request.

EMR config

Terminated by user request, which is not true

This is the configuration I have defined in airflow:

JOB_FLOW_OVERRIDES = {
    "Name": "X",
    "Applications": [
        {"Name": "Spark"},
        {"Name": "Ganglia"},
        ],
    "Instances": {
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        "Ec2SubnetId":"X",
        "EmrManagedMasterSecurityGroup": "X",
        "EmrManagedSlaveSecurityGroup": "X",
        "ServiceAccessSecurityGroup": "X",
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "SPOT",
                "InstanceRole": "MASTER",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 1,
            },
            {
                "Name": "Core nodes",
                "Market": "SPOT",
                "InstanceRole": "CORE",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 2,
            }
        ],
        "Ec2KeyName": "X",
    },
    "LogUri": "X",
    "ReleaseLabel": "X",
    "VisibleToAllUsers": True,
    "JobFlowRole": "DataPipelineDefaultResourceRole",
    "ServiceRole": "DataPipelineDefaultRole",
}
已提問 5 個月前檢視次數 151 次
1 個回答
3

Hello,

Please check the CloudTrail and see which user initiates the TerminateJobFlows API call. There could be the user or the Org policy who gain access to terminate the cluster would be performed this. For an instance, if any policy that created in your Org when the cluster created using public subnet will be terminated automatically. You can get more insights from referring the CloudTrail. If the TerminateJobFlows triggered by Airflow role, then it is handled as part of DAG execution.

AWS
支援工程師
已回答 5 個月前
profile picture
專家
已審閱 5 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南