How do I use Step Functions to create EMR clusters with different specifications?

Question

I'm using Step Functions to create transient Amazon EMR clusters. For the task node, I use a Spot instance, and for timeouts, I have set up a rule to terminate the cluster. Can I use multiple spot rules in the Step Function script for creating clusters with different specifications (example: i3.8xlarge and i4.8xlarge)? Using the Amazon EMR console, its possible to mention up to five instance types in the instance fleet. Is there a sample script that can help me test with multiple instance types?
I'm referencing [Call Amazon EMR with Step Functions][1] to create my cluster.
[1]: https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html

Behram · Accepted Answer

Add the JSON code similar to the following in the Step Function:

```
[
  {
    "Name": "MainFleet",
    "InstanceFleetType": "MAIN",
    "TargetOnDemandCapacity": 1,
    "InstanceTypeConfigs": [{"InstanceType": "m3.xlarge"}]
  },
  {
    "Name": "CoreFleet",
    "InstanceFleetType": "CORE",
    "TargetSpotCapacity": 11,
    "TargetOnDemandCapacity": 11,
    "LaunchSpecifications": {
      "SpotSpecification": {
        "TimeoutDurationMinutes": 20,
        "TimeoutAction": "SWITCH_TO_ON_DEMAND"
      }
    },
    "InstanceTypeConfigs": [
      {
        "InstanceType": "r4.xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 1
      },
      {
        "InstanceType": "r4.2xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 2
      },
      {
        "InstanceType": "r4.4xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 4
      }
    ]
  }
]
```

How do I use Step Functions to create EMR clusters with different specifications?

Relevant content