AWS CDK - AWS EMR (creating task instances)

0

Hello,

I was trying to create emr cluster using aws_cdk.aws_emr module, but I could not figure out a way to add task instances. It is possible using aws_cdk.aws_stepfunctions_tasks, but then it does not have manged_scaling_policy as available in aws_cdk.aws_emr .

If anyone here could point me towards a way to achieve it, I would really appreciate it.

Thank you, Manjil

Manjil
demandé il y a 2 ans850 vues
2 réponses
0

Hello,

You can use aws_cdk.aws_emr property "JobFlowInstancesConfigProperty" for instantiating task instances. Please see below example and hope this helps.

``

eg:
emr_clst = CfnCluster(self,"my_emr_cluster",
            name="my-test-cluster",
            applications=[CfnCluster.ApplicationProperty(name="Hive")],
            log_uri=f"s3://mybucket/emr/",
            release_label=<<emr_version>>,
            visible_to_all_users=True,
            service_role=<<service_role>>
            job_flow_role=<<job_flow_role>>,
            instances=CfnCluster.JobFlowInstancesConfigProperty(
                termination_protected=False,
                master_instance_group=CfnCluster.InstanceGroupConfigProperty(
                    instance_count=1, 
                    instance_type="c5.9xlarge"
                ),
                core_instance_group=CfnCluster.InstanceGroupConfigProperty(
                    instance_count=5, 
                    instance_type="c5.9xlarge"
                ),
                ec2_subnet_id=<<subnet_id>>
            ),
            configurations=[
                CfnCluster.ConfigurationProperty(
                    ...
                    ...
                    ...
                )
            ]
        )

``

Please replace the items in << >> with the respecting values and refer the documentation for more details- https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-emr.CfnCluster.JobFlowInstancesConfigProperty.html.

AWS
répondu il y a 2 ans
  • Thank you for your answer. I also reached to this point, but I am not really sure on what goes inside CfnCluster.ConfigurationProperty?

0

These are the configuration properties that you want to add it to the EMR cluster. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.

Please refer the link for more details on ConfigurationProperty - https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-emr.CfnCluster.ConfigurationProperty.html

AWS
répondu il y a 2 ans
  • Thank you for pointing it out, but could you please provide with a short example on what the classification would be, what to put in the properties? Because, I did not find it in the document. May be you could point me towards it too.

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions