I have created trivial step function to add a step to an EMR cluster.
The jobflowid is not being recognised.
Does anyone have a working solution here? Or pointer to what set up may be missing?
The state machine:
{
"StartAt": "add_emr_step",
"States": {
"add_emr_step": {
"End": true,
"InputPath": "$",
"Parameters": {
"ClusterId": "j-INTERESTINGID",
"Step": {
"Name": "$$.Execution.Id",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
]
}
}
},
"Type": "Task",
"Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
"ResultPath": "$.guid"
}
},
"TimeoutSeconds": 30
}
Even with admin permissions the following error is returning.
The EMR cluster is marked as visible.
Copying and pasting the jobflowid into the console, it shows as expected.
Error
EMR.AmazonElasticMapReduceException
Cause
Specified job flow ID does not exist. (Service:
AmazonElasticMapReduce; Status Code: 400; Error Code:
ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)
The IAM Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"elasticmapreduce:AddJobFlowSteps",
"elasticmapreduce:DescribeStep",
"elasticmapreduce:CancelSteps"
],
"Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
"Effect": "Allow"
},
{
"Action": [
"events:PutTargets",
"events:PutRule",
"events:DescribeRule"
],
"Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
"Effect": "Allow"
}
]
}
The API call in CloudTrail
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
"arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
"accountId": "accountid",
"accessKeyId": "ASIA36BAFPT6UG3XKKHI",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROA36BAFPT62RGX2RMVQ",
"arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
"accountId": "accountid",
"userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2020-06-08T23:29:29Z"
}
},
"invokedBy": "states.amazonaws.com"
},
"eventTime": "2020-06-08T23:29:29Z",
"eventSource": "elasticmapreduce.amazonaws.com",
"eventName": "AddJobFlowSteps",
"awsRegion": "us-east-1",
"sourceIPAddress": "states.amazonaws.com",
"userAgent": "states.amazonaws.com",
"errorCode": "ValidationException",
"errorMessage": "Specified job flow ID does not exist.",
"requestParameters": {
"jobFlowId": "j-288AS5TZY5CY7",
"steps": [
{
"name": "$$.Execution.Id",
"actionOnFailure": "CONTINUE",
"hadoopJarStep": {
"jar": "command-runner.jar",
"args": []
}
}
]
},
"responseElements": null,
"requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
"eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
"eventType": "AwsApiCall",
"recipientAccountId": "accountid"
}