How do I use the AWSSupport-AnalyzeEMRLogs runbook to identify errors when I run an Amazon EMR job on a cluster?

Short description

Use the AWSSupport-AnalyzeEMRLogs runbook to identify errors when you run an Amazon EMR job on a cluster. The runbook searches for predefined errors and patterns in logs. Log entries are used to create Amazon CloudWatch Events so that you can take actions as required. Also, the runbook publishes log entries to the Amazon CloudWatch Logs log group that you choose.

Note: Instances associated with the ClusterID that you specify must be managed by AWS Systems Manager. You have the options to run the automation manually or on a scheduled time interval, or you can remove a schedule. The runbook supports Amazon EMR release versions 5.20 to 6.30.

Resolution

Prerequisites

Before you run the automation, make sure that the AWS Identity and Access Management (IAM) user or role that you're using has the required IAM permissions.

Run the Systems Manager automation runbook

1. Open the AWS Systems Manager console.
2. In the navigation pane, choose Documents.
3. In the search bar, enter AWSSupport-AnalyzeEMRLogs.
4. Select the AWSSupport-AnalyzeEMRLogs document and choose Execute automation or Run this automation (console).
5. For the input parameters, enter the following:
   • ClusterID (required): The ID of the cluster that has the nodes logs that you want to analyze.
   • AutomationAssumeRole (optional): The ARN of the role that allows the automation to perform the actions on your behalf.
   • Operation (required): The operation to perform on the cluster. Valid values are Run Once, Schedule, Remove Schedule.
   • IntervalTime (optional): The duration of time between running the automation. This parameter is applicable if you specify Schedule for the Operation parameter.
   • LogToCloudWatchLogs (optional): If you specify yes for this parameter, then the automation creates an Amazon CloudWatch Logs log group with the name that you specified. Valid values are yes or no.
   • CloudWatchLogGroup (optional): The name of the Amazon CloudWatch Logs log group that you want to store matched log entries in. This parameter is applicable if you specify yes for the LogToCloudWatchLogs parameter.
   • CreateLogInsightsDashboard (optional): If you specify yes for this parameter, then an Amazon CloudWatch dashboard is created if it doesn't already exist. This parameter is applicable if you specify yes for the LogToCloudWatchLogs parameter. Valid values are yes or no.
   • CreateMetricFilters (optional): If you specify yes for this parameter, then metric filters are created for the Amazon CloudWatch Logs log group. This parameter is applicable if you specify yes for the LogToCloudWatchLogs parameter. Valid values are yes or no.
6. Choose Execute.
7. When the automation is completed, review the Outputs section for the log location information.

Related information

Run an automation

Setting up automation

Systems Manager automation runbook reference