AWS Step Function does not recognize job flow ID

1

I have created trivial step function to add a step to an EMR cluster.
The jobflowid is not being recognised.
Does anyone have a working solution here? Or pointer to what set up may be missing?

The state machine:

{
  "StartAt": "add_emr_step",
  "States": {
    "add_emr_step": {
      "End": true,
      "InputPath": "$",
      "Parameters": {
        "ClusterId": "j-INTERESTINGID",
        "Step": {
          "Name": "$$.Execution.Id",
          "ActionOnFailure": "CONTINUE",
          "HadoopJarStep": {
            "Jar": "command-runner.jar",
            "Args": [
              "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
            ]
          }
        }
      },
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
      "ResultPath": "$.guid"
    }
  },
  "TimeoutSeconds": 30
}

Even with admin permissions the following error is returning.
The EMR cluster is marked as visible.
Copying and pasting the jobflowid into the console, it shows as expected.

Error

EMR.AmazonElasticMapReduceException

Cause

Specified job flow ID does not exist. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)

The IAM Role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "events:PutTargets",
                "events:PutRule",
                "events:DescribeRule"
            ],
            "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
            "Effect": "Allow"
        }
    ]
}

The API call in CloudTrail

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "accountId": "accountid",
        "accessKeyId": "ASIA36BAFPT6UG3XKKHI",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA36BAFPT62RGX2RMVQ",
                "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
                "accountId": "accountid",
                "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-06-08T23:29:29Z"
            }
        },
        "invokedBy": "states.amazonaws.com"
    },
    "eventTime": "2020-06-08T23:29:29Z",
    "eventSource": "elasticmapreduce.amazonaws.com",
    "eventName": "AddJobFlowSteps",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "states.amazonaws.com",
    "userAgent": "states.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "Specified job flow ID does not exist.",
    "requestParameters": {
        "jobFlowId": "j-288AS5TZY5CY7",
        "steps": [
            {
                "name": "$$.Execution.Id",
                "actionOnFailure": "CONTINUE",
                "hadoopJarStep": {
                    "jar": "command-runner.jar",
                    "args": []
                }
            }
        ]
    },
    "responseElements": null,
    "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
    "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
    "eventType": "AwsApiCall",
    "recipientAccountId": "accountid"
}
AWS
fmcmac
preguntada hace 4 años1325 visualizaciones
1 Respuesta
1
Respuesta aceptada

Try this.

 "ClusterId.$": "j-INTERESTINGID",

Update : Please see if the step function role has the following

 {
            "Action": [
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticmapreduce:*"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:<region>:<account>:cluster/*"
            ],
            "Effect": "Allow"
        }
AWS
EXPERTO
respondido hace 4 años

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas