AWS CDK - AWS EMR (creating task instances)

0

Hello,

I was trying to create emr cluster using aws_cdk.aws_emr module, but I could not figure out a way to add task instances. It is possible using aws_cdk.aws_stepfunctions_tasks, but then it does not have manged_scaling_policy as available in aws_cdk.aws_emr .

If anyone here could point me towards a way to achieve it, I would really appreciate it.

Thank you, Manjil

Manjil
preguntada hace 2 años850 visualizaciones
2 Respuestas
0

Hello,

You can use aws_cdk.aws_emr property "JobFlowInstancesConfigProperty" for instantiating task instances. Please see below example and hope this helps.

``

eg:
emr_clst = CfnCluster(self,"my_emr_cluster",
            name="my-test-cluster",
            applications=[CfnCluster.ApplicationProperty(name="Hive")],
            log_uri=f"s3://mybucket/emr/",
            release_label=<<emr_version>>,
            visible_to_all_users=True,
            service_role=<<service_role>>
            job_flow_role=<<job_flow_role>>,
            instances=CfnCluster.JobFlowInstancesConfigProperty(
                termination_protected=False,
                master_instance_group=CfnCluster.InstanceGroupConfigProperty(
                    instance_count=1, 
                    instance_type="c5.9xlarge"
                ),
                core_instance_group=CfnCluster.InstanceGroupConfigProperty(
                    instance_count=5, 
                    instance_type="c5.9xlarge"
                ),
                ec2_subnet_id=<<subnet_id>>
            ),
            configurations=[
                CfnCluster.ConfigurationProperty(
                    ...
                    ...
                    ...
                )
            ]
        )

``

Please replace the items in << >> with the respecting values and refer the documentation for more details- https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-emr.CfnCluster.JobFlowInstancesConfigProperty.html.

AWS
respondido hace 2 años
  • Thank you for your answer. I also reached to this point, but I am not really sure on what goes inside CfnCluster.ConfigurationProperty?

0

These are the configuration properties that you want to add it to the EMR cluster. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.

Please refer the link for more details on ConfigurationProperty - https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-emr.CfnCluster.ConfigurationProperty.html

AWS
respondido hace 2 años
  • Thank you for pointing it out, but could you please provide with a short example on what the classification would be, what to put in the properties? Because, I did not find it in the document. May be you could point me towards it too.

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas