How to add logging in step function configuration for EMR serverless Job

Question

I am executing pyspark script on EMR serverless and i am using AWS step function for creating EMR application and submitting job in it. I want to know how to add logging option in AWS step function configuration while creating EMR application or submitting job in it so that i can see log files from spark job run in cloudwatch or in s3. this is my current block from StartJobRun state from AWS step function configuration.

"EMR Serverless StartJobRun": 
{ "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:startJobRun", "Parameters": 
{ "ApplicationId.$": "$.ApplicationId", 
"ExecutionRoleArn": "arn:aws:iam:/demo-emrserverless-iam-role",
 "JobDriver": 
{ "SparkSubmit": 
{ "EntryPoint": "s3://demo/test.py",
 "EntryPointArguments.$":[], 
"SparkSubmitParameters": "--conf } },

i tried to add "JobConfiguration" but it is not accepting JobConfiguration

Answer

Hello,

In order to add logging to EMR serverless job using the API StartJobRUn[[1]](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html). You need to use the key "ConfigurationOverrides"[[2]](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_ConfigurationOverrides.html) and then "MonitoringConfiguration"[[3]](https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_MonitoringConfiguration.html) inside it to enable S3 logging or cloudwatch logging etc.

More information about logging in EMR Serverless can be found here[[4]](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/logging-monitoring.html).

How to add logging in step function configuration for EMR serverless Job

Relevanter Inhalt