EMR Auto scaling with step function

Hello Team,

[+] I understand you are facing issues while launching an Amazon EMR cluster with the 'AutoScalingPolicy' in Step Functions.

➤ Upon checking internally, I have verified that we can achieve this use-case by leveraging createCluster.sync task.

➙ Below is the sample script you can use to create an EMR cluster with custom autoscaling in Step function :

{
 "Comment": "A Step Function to create an EMR cluster with custom autoscaling",
 "StartAt": "Create EMR Cluster",
 "States": {
  "Create EMR Cluster": {
   "Type": "Task",
   "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
   "Parameters": {
    "Name": "MyEMRCluster",
    "ReleaseLabel": "emr-6.5.0",
    "Applications": [
     {
      "Name": "Spark"
     }
    ],
    "Instances": {
     "InstanceGroups": [
{
       "Name": "Master",
       "InstanceRole": "MASTER",
       "InstanceType": "m5.xlarge",
       "InstanceCount": 1
      },
      {
       "Name": "Core",
       "InstanceRole": "CORE",
       "InstanceType": "m5.xlarge",
       "InstanceCount": 2
      }
     ],
"KeepJobFlowAliveWhenNoSteps": true,
     "TerminationProtected": false,
     "Ec2KeyName": “key_name,
     "Ec2SubnetId": “subnetID”
    },
    "ManagedScalingPolicy": {
     "ComputeLimits": {
      "UnitType": "Instances",
      "MinimumCapacityUnits": 2,
      "MaximumCapacityUnits": 10,
      "MaximumOnDemandCapacityUnits": 10,
      "MaximumCoreCapacityUnits": 10
     }
    },
"VisibleToAllUsers": true,
    "JobFlowRole": "EMR_EC2_DefaultRole",
    "ServiceRole": "EMR_DefaultRole",
    "AutoScalingRole": "EMR_AutoScaling_DefaultRole"
   },
   "End": true
  }
 }
}

Please feel free to reach out for any further queries or concerns and I'll be happy to assist.

Thank you, and I hope you have a great day!

References:

[1] https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html

[2] https://aws.amazon.com/blogs/aws/new-using-step-functions-to-orchestrate-amazon-emr-workloads/

Based on the information available, it appears that using 'AutoScalingPolicy' directly within a Step Functions state machine for launching an EMR cluster is not supported. The error message you received suggests that the 'AutoScalingPolicy' field is not recognized by Step Functions when integrating with EMR.

However, there are alternative approaches you can consider to achieve auto-scaling for your EMR cluster using Step Functions:

1. Use the Application Auto Scaling service: Instead of specifying the 'AutoScalingPolicy' in the cluster creation step, you can create a separate step in your Step Functions workflow to configure auto-scaling using Application Auto Scaling. This service supports EMR clusters and can be integrated with Step Functions.

2. Implement step scaling policies: You can create step scaling policies for your EMR cluster using Application Auto Scaling. These policies allow you to define scaling adjustments based on CloudWatch alarms. You can incorporate the creation of these policies into your Step Functions workflow.

3. Use the EMR service integration APIs: Step Functions provides optimized integration with EMR, including APIs for creating and managing EMR clusters. While these APIs don't directly support 'AutoScalingPolicy', you can use them to create the cluster and then add additional steps to configure auto-scaling separately.

4. Split the process: Create the EMR cluster without auto-scaling using Step Functions, and then use a subsequent step to apply the auto-scaling policy to the cluster using the appropriate EMR or Application Auto Scaling API calls.

It's important to note that while Step Functions may not directly support the 'AutoScalingPolicy' field in the cluster creation JSON, you can still achieve auto-scaling functionality by breaking down the process into multiple steps within your Step Functions workflow. This approach allows you to leverage the capabilities of both EMR and Application Auto Scaling while working within the constraints of Step Functions.
Sources
Step scaling policies for Application Auto Scaling - Application Auto Scaling
How step scaling for Application Auto Scaling works - Application Auto Scaling
Create and manage Amazon EMR clusters with Step Functions - AWS Step Functions