AWS CDK - AWS EMR (creating task instances)

0

Hello,

I was trying to create emr cluster using aws_cdk.aws_emr module, but I could not figure out a way to add task instances. It is possible using aws_cdk.aws_stepfunctions_tasks, but then it does not have manged_scaling_policy as available in aws_cdk.aws_emr .

If anyone here could point me towards a way to achieve it, I would really appreciate it.

Thank you, Manjil

Manjil
asked 2 years ago808 views
2 Answers
0

Hello,

You can use aws_cdk.aws_emr property "JobFlowInstancesConfigProperty" for instantiating task instances. Please see below example and hope this helps.

``

eg:
emr_clst = CfnCluster(self,"my_emr_cluster",
            name="my-test-cluster",
            applications=[CfnCluster.ApplicationProperty(name="Hive")],
            log_uri=f"s3://mybucket/emr/",
            release_label=<<emr_version>>,
            visible_to_all_users=True,
            service_role=<<service_role>>
            job_flow_role=<<job_flow_role>>,
            instances=CfnCluster.JobFlowInstancesConfigProperty(
                termination_protected=False,
                master_instance_group=CfnCluster.InstanceGroupConfigProperty(
                    instance_count=1, 
                    instance_type="c5.9xlarge"
                ),
                core_instance_group=CfnCluster.InstanceGroupConfigProperty(
                    instance_count=5, 
                    instance_type="c5.9xlarge"
                ),
                ec2_subnet_id=<<subnet_id>>
            ),
            configurations=[
                CfnCluster.ConfigurationProperty(
                    ...
                    ...
                    ...
                )
            ]
        )

``

Please replace the items in << >> with the respecting values and refer the documentation for more details- https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-emr.CfnCluster.JobFlowInstancesConfigProperty.html.

AWS
answered 2 years ago
  • Thank you for your answer. I also reached to this point, but I am not really sure on what goes inside CfnCluster.ConfigurationProperty?

0

These are the configuration properties that you want to add it to the EMR cluster. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.

Please refer the link for more details on ConfigurationProperty - https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-emr.CfnCluster.ConfigurationProperty.html

AWS
answered 2 years ago
  • Thank you for pointing it out, but could you please provide with a short example on what the classification would be, what to put in the properties? Because, I did not find it in the document. May be you could point me towards it too.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions