How to add logging in step function configuration for EMR serverless Job

0

I am executing pyspark script on EMR serverless and i am using AWS step function for creating EMR application and submitting job in it. I want to know how to add logging option in AWS step function configuration while creating EMR application or submitting job in it so that i can see log files from spark job run in cloudwatch or in s3. this is my current block from StartJobRun state from AWS step function configuration.

"EMR Serverless StartJobRun": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:startJobRun", "Parameters": { "ApplicationId.$": "$.ApplicationId", "ExecutionRoleArn": "arn:aws:iam:/demo-emrserverless-iam-role", "JobDriver": { "SparkSubmit": { "EntryPoint": "s3://demo/test.py", "EntryPointArguments.$":[], "SparkSubmitParameters": "--conf } },

i tried to add "JobConfiguration" but it is not accepting JobConfiguration

1回答
0

Hello,

In order to add logging to EMR serverless job using the API StartJobRUn[1]. You need to use the key "ConfigurationOverrides"[2] and then "MonitoringConfiguration"[3] inside it to enable S3 logging or cloudwatch logging etc.

More information about logging in EMR Serverless can be found here[4].

AWS
回答済み 19日前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ