EMR automatically finishing when manually finishing is selected via Airflow

0

So I define manually finishing using the RunJobFlow operator (https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) "KeepJobFlowAliveWhenNoSteps": True. However, the cluster cluster automatically finishes but says terminated by user request.

EMR config

Terminated by user request, which is not true

This is the configuration I have defined in airflow:

JOB_FLOW_OVERRIDES = {
    "Name": "X",
    "Applications": [
        {"Name": "Spark"},
        {"Name": "Ganglia"},
        ],
    "Instances": {
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        "Ec2SubnetId":"X",
        "EmrManagedMasterSecurityGroup": "X",
        "EmrManagedSlaveSecurityGroup": "X",
        "ServiceAccessSecurityGroup": "X",
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "SPOT",
                "InstanceRole": "MASTER",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 1,
            },
            {
                "Name": "Core nodes",
                "Market": "SPOT",
                "InstanceRole": "CORE",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 2,
            }
        ],
        "Ec2KeyName": "X",
    },
    "LogUri": "X",
    "ReleaseLabel": "X",
    "VisibleToAllUsers": True,
    "JobFlowRole": "DataPipelineDefaultResourceRole",
    "ServiceRole": "DataPipelineDefaultRole",
}
질문됨 5달 전151회 조회
1개 답변
3

Hello,

Please check the CloudTrail and see which user initiates the TerminateJobFlows API call. There could be the user or the Org policy who gain access to terminate the cluster would be performed this. For an instance, if any policy that created in your Org when the cluster created using public subnet will be terminated automatically. You can get more insights from referring the CloudTrail. If the TerminateJobFlows triggered by Airflow role, then it is handled as part of DAG execution.

AWS
지원 엔지니어
답변함 5달 전
profile picture
전문가
검토됨 5달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠