AWS Step Function does not recognize job flow ID

1

I have created trivial step function to add a step to an EMR cluster.
The jobflowid is not being recognised.
Does anyone have a working solution here? Or pointer to what set up may be missing?

The state machine:

{
  "StartAt": "add_emr_step",
  "States": {
    "add_emr_step": {
      "End": true,
      "InputPath": "$",
      "Parameters": {
        "ClusterId": "j-INTERESTINGID",
        "Step": {
          "Name": "$$.Execution.Id",
          "ActionOnFailure": "CONTINUE",
          "HadoopJarStep": {
            "Jar": "command-runner.jar",
            "Args": [
              "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
            ]
          }
        }
      },
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
      "ResultPath": "$.guid"
    }
  },
  "TimeoutSeconds": 30
}

Even with admin permissions the following error is returning.
The EMR cluster is marked as visible.
Copying and pasting the jobflowid into the console, it shows as expected.

Error

EMR.AmazonElasticMapReduceException

Cause

Specified job flow ID does not exist. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)

The IAM Role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "events:PutTargets",
                "events:PutRule",
                "events:DescribeRule"
            ],
            "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
            "Effect": "Allow"
        }
    ]
}

The API call in CloudTrail

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "accountId": "accountid",
        "accessKeyId": "ASIA36BAFPT6UG3XKKHI",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA36BAFPT62RGX2RMVQ",
                "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
                "accountId": "accountid",
                "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-06-08T23:29:29Z"
            }
        },
        "invokedBy": "states.amazonaws.com"
    },
    "eventTime": "2020-06-08T23:29:29Z",
    "eventSource": "elasticmapreduce.amazonaws.com",
    "eventName": "AddJobFlowSteps",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "states.amazonaws.com",
    "userAgent": "states.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "Specified job flow ID does not exist.",
    "requestParameters": {
        "jobFlowId": "j-288AS5TZY5CY7",
        "steps": [
            {
                "name": "$$.Execution.Id",
                "actionOnFailure": "CONTINUE",
                "hadoopJarStep": {
                    "jar": "command-runner.jar",
                    "args": []
                }
            }
        ]
    },
    "responseElements": null,
    "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
    "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
    "eventType": "AwsApiCall",
    "recipientAccountId": "accountid"
}
AWS
fmcmac
已提問 4 年前檢視次數 1325 次
1 個回答
1
已接受的答案

Try this.

 "ClusterId.$": "j-INTERESTINGID",

Update : Please see if the step function role has the following

 {
            "Action": [
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticmapreduce:*"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:<region>:<account>:cluster/*"
            ],
            "Effect": "Allow"
        }
AWS
專家
已回答 4 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南