AWS Step Function does not recognize job flow ID

1

I have created trivial step function to add a step to an EMR cluster.
The jobflowid is not being recognised.
Does anyone have a working solution here? Or pointer to what set up may be missing?

The state machine:

{
  "StartAt": "add_emr_step",
  "States": {
    "add_emr_step": {
      "End": true,
      "InputPath": "$",
      "Parameters": {
        "ClusterId": "j-INTERESTINGID",
        "Step": {
          "Name": "$$.Execution.Id",
          "ActionOnFailure": "CONTINUE",
          "HadoopJarStep": {
            "Jar": "command-runner.jar",
            "Args": [
              "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
            ]
          }
        }
      },
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
      "ResultPath": "$.guid"
    }
  },
  "TimeoutSeconds": 30
}

Even with admin permissions the following error is returning.
The EMR cluster is marked as visible.
Copying and pasting the jobflowid into the console, it shows as expected.

Error

EMR.AmazonElasticMapReduceException

Cause

Specified job flow ID does not exist. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)

The IAM Role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "events:PutTargets",
                "events:PutRule",
                "events:DescribeRule"
            ],
            "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
            "Effect": "Allow"
        }
    ]
}

The API call in CloudTrail

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "accountId": "accountid",
        "accessKeyId": "ASIA36BAFPT6UG3XKKHI",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA36BAFPT62RGX2RMVQ",
                "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
                "accountId": "accountid",
                "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-06-08T23:29:29Z"
            }
        },
        "invokedBy": "states.amazonaws.com"
    },
    "eventTime": "2020-06-08T23:29:29Z",
    "eventSource": "elasticmapreduce.amazonaws.com",
    "eventName": "AddJobFlowSteps",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "states.amazonaws.com",
    "userAgent": "states.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "Specified job flow ID does not exist.",
    "requestParameters": {
        "jobFlowId": "j-288AS5TZY5CY7",
        "steps": [
            {
                "name": "$$.Execution.Id",
                "actionOnFailure": "CONTINUE",
                "hadoopJarStep": {
                    "jar": "command-runner.jar",
                    "args": []
                }
            }
        ]
    },
    "responseElements": null,
    "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
    "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
    "eventType": "AwsApiCall",
    "recipientAccountId": "accountid"
}
AWS
fmcmac
gefragt vor 4 Jahren1325 Aufrufe
1 Antwort
1
Akzeptierte Antwort

Try this.

 "ClusterId.$": "j-INTERESTINGID",

Update : Please see if the step function role has the following

 {
            "Action": [
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticmapreduce:*"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:<region>:<account>:cluster/*"
            ],
            "Effect": "Allow"
        }
AWS
EXPERTE
beantwortet vor 4 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen