AWS Step Function does not recognize job flow ID

1

I have created trivial step function to add a step to an EMR cluster.
The jobflowid is not being recognised.
Does anyone have a working solution here? Or pointer to what set up may be missing?

The state machine:

{
  "StartAt": "add_emr_step",
  "States": {
    "add_emr_step": {
      "End": true,
      "InputPath": "$",
      "Parameters": {
        "ClusterId": "j-INTERESTINGID",
        "Step": {
          "Name": "$$.Execution.Id",
          "ActionOnFailure": "CONTINUE",
          "HadoopJarStep": {
            "Jar": "command-runner.jar",
            "Args": [
              "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
            ]
          }
        }
      },
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
      "ResultPath": "$.guid"
    }
  },
  "TimeoutSeconds": 30
}

Even with admin permissions the following error is returning.
The EMR cluster is marked as visible.
Copying and pasting the jobflowid into the console, it shows as expected.

Error

EMR.AmazonElasticMapReduceException

Cause

Specified job flow ID does not exist. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)

The IAM Role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "events:PutTargets",
                "events:PutRule",
                "events:DescribeRule"
            ],
            "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
            "Effect": "Allow"
        }
    ]
}

The API call in CloudTrail

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "accountId": "accountid",
        "accessKeyId": "ASIA36BAFPT6UG3XKKHI",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA36BAFPT62RGX2RMVQ",
                "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
                "accountId": "accountid",
                "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-06-08T23:29:29Z"
            }
        },
        "invokedBy": "states.amazonaws.com"
    },
    "eventTime": "2020-06-08T23:29:29Z",
    "eventSource": "elasticmapreduce.amazonaws.com",
    "eventName": "AddJobFlowSteps",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "states.amazonaws.com",
    "userAgent": "states.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "Specified job flow ID does not exist.",
    "requestParameters": {
        "jobFlowId": "j-288AS5TZY5CY7",
        "steps": [
            {
                "name": "$$.Execution.Id",
                "actionOnFailure": "CONTINUE",
                "hadoopJarStep": {
                    "jar": "command-runner.jar",
                    "args": []
                }
            }
        ]
    },
    "responseElements": null,
    "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
    "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
    "eventType": "AwsApiCall",
    "recipientAccountId": "accountid"
}
AWS
fmcmac
已提问 4 年前1325 查看次数
1 回答
1
已接受的回答

Try this.

 "ClusterId.$": "j-INTERESTINGID",

Update : Please see if the step function role has the following

 {
            "Action": [
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticmapreduce:*"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:<region>:<account>:cluster/*"
            ],
            "Effect": "Allow"
        }
AWS
专家
已回答 4 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则