EMR automatically finishing when manually finishing is selected via Airflow

0

So I define manually finishing using the RunJobFlow operator (https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html) "KeepJobFlowAliveWhenNoSteps": True. However, the cluster cluster automatically finishes but says terminated by user request.

EMR config

Terminated by user request, which is not true

This is the configuration I have defined in airflow:

JOB_FLOW_OVERRIDES = {
    "Name": "X",
    "Applications": [
        {"Name": "Spark"},
        {"Name": "Ganglia"},
        ],
    "Instances": {
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        "Ec2SubnetId":"X",
        "EmrManagedMasterSecurityGroup": "X",
        "EmrManagedSlaveSecurityGroup": "X",
        "ServiceAccessSecurityGroup": "X",
        "InstanceGroups": [
            {
                "Name": "Master nodes",
                "Market": "SPOT",
                "InstanceRole": "MASTER",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 1,
            },
            {
                "Name": "Core nodes",
                "Market": "SPOT",
                "InstanceRole": "CORE",
                "InstanceType": "r4.xlarge",
                "InstanceCount": 2,
            }
        ],
        "Ec2KeyName": "X",
    },
    "LogUri": "X",
    "ReleaseLabel": "X",
    "VisibleToAllUsers": True,
    "JobFlowRole": "DataPipelineDefaultResourceRole",
    "ServiceRole": "DataPipelineDefaultRole",
}
feita há 5 meses151 visualizações
1 Resposta
3

Hello,

Please check the CloudTrail and see which user initiates the TerminateJobFlows API call. There could be the user or the Org policy who gain access to terminate the cluster would be performed this. For an instance, if any policy that created in your Org when the cluster created using public subnet will be terminated automatically. You can get more insights from referring the CloudTrail. If the TerminateJobFlows triggered by Airflow role, then it is handled as part of DAG execution.

AWS
ENGENHEIRO DE SUPORTE
respondido há 5 meses
profile picture
ESPECIALISTA
avaliado há 5 meses

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas