EMR automatically finishing when manually finishing is selected via Airflow

0

So I define manually finishing using the RunJobFlow operator (https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) "KeepJobFlowAliveWhenNoSteps": True. However, the cluster cluster automatically finishes but says terminated by user request.

EMR config

Terminated by user request, which is not true

This is the configuration I have defined in airflow:

JOB_FLOW_OVERRIDES = {
    "Name": "X",
    "Applications": [
        {"Name": "Spark"},
        {"Name": "Ganglia"},
        ],
    "Instances": {
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        "Ec2SubnetId":"X",
        "EmrManagedMasterSecurityGroup": "X",
        "EmrManagedSlaveSecurityGroup": "X",
        "ServiceAccessSecurityGroup": "X",
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "SPOT",
                "InstanceRole": "MASTER",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 1,
            },
            {
                "Name": "Core nodes",
                "Market": "SPOT",
                "InstanceRole": "CORE",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 2,
            }
        ],
        "Ec2KeyName": "X",
    },
    "LogUri": "X",
    "ReleaseLabel": "X",
    "VisibleToAllUsers": True,
    "JobFlowRole": "DataPipelineDefaultResourceRole",
    "ServiceRole": "DataPipelineDefaultRole",
}
demandé il y a 5 mois151 vues
1 réponse
3

Hello,

Please check the CloudTrail and see which user initiates the TerminateJobFlows API call. There could be the user or the Org policy who gain access to terminate the cluster would be performed this. For an instance, if any policy that created in your Org when the cluster created using public subnet will be terminated automatically. You can get more insights from referring the CloudTrail. If the TerminateJobFlows triggered by Airflow role, then it is handled as part of DAG execution.

AWS
INGÉNIEUR EN ASSISTANCE TECHNIQUE
répondu il y a 5 mois
profile picture
EXPERT
vérifié il y a 5 mois

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions