By using AWS re:Post, you agree to the Terms of Use
/How do I use Step Functions to create EMR clusters with different specifications?/

How do I use Step Functions to create EMR clusters with different specifications?

0

I'm using Step Functions to create transient Amazon EMR clusters. For the task node, I use a Spot instance, and for timeouts, I have set up a rule to terminate the cluster. Can I use multiple spot rules in the Step Function script for creating clusters with different specifications (example: i3.8xlarge and i4.8xlarge)? Using the Amazon EMR console, its possible to mention up to five instance types in the instance fleet. Is there a sample script that can help me test with multiple instance types? I'm referencing Call Amazon EMR with Step Functions to create my cluster.

1 Answers
0
Accepted Answer

Add the JSON code similar to the following in the Step Function:

[
{
  "Name": "MainFleet",
  "InstanceFleetType": "MAIN",
  "TargetOnDemandCapacity": 1,
  "InstanceTypeConfigs": [{"InstanceType": "m3.xlarge"}]
},
{
  "Name": "CoreFleet",
  "InstanceFleetType": "CORE",
  "TargetSpotCapacity": 11,
  "TargetOnDemandCapacity": 11,
  "LaunchSpecifications": {
    "SpotSpecification": {
      "TimeoutDurationMinutes": 20,
      "TimeoutAction": "SWITCH_TO_ON_DEMAND"
    }
  },
  "InstanceTypeConfigs": [
    {
      "InstanceType": "r4.xlarge",
      "BidPriceAsPercentageOfOnDemandPrice": 50,
      "WeightedCapacity": 1
    },
    {
      "InstanceType": "r4.2xlarge",
      "BidPriceAsPercentageOfOnDemandPrice": 50,
      "WeightedCapacity": 2
    },
    {
      "InstanceType": "r4.4xlarge",
      "BidPriceAsPercentageOfOnDemandPrice": 50,
      "WeightedCapacity": 4
    }
  ]
}
]
EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions