Improving critical workload monitoring with New Relic integration

Introduction

AWS Incident Detection and Response provides proactive and context aware incident management for onboarded business critical workloads with 5-minute responses from experienced Incident Management Engineers (IMEs). This process helps you to reduce the mean time to resolve incidents and improve observability. IMEs provide 24/7 monitoring to detect critical incidents.

You might use Application Performance Monitoring (APM) tools, such as Datadog, New Relic, Dynatrace, and Splunk, as part of your observability technical stack. Incident Detection and Response supports alarm ingestion from these APM tools using Amazon EventBridge. Some APM tools directly integrate with EventBridge. For a complete list of APMs, see Amazon EventBridge integrations. A few other tools use webhooks to ingest alarms from APM. This article covers the following topics:

• Configure new relic to integrate with AWS

• Set up the partner event bus to bridge New Relic with AWS

• Payload transformation using AWS Lambda

• Set up the integration with Incident Detection and Response for the critical alerts to be automatically sent to Incident Detection and Response for investigation by the AWS Support team.

Architecture overview

This solution shows how you can integrate New Relic with Incident Detection and Response. When New Relic detects an alert, it routes the data through EventBridge. Then, it generates an AWS Support case automatically to Incident Detection and Response for further investigation with relevant information. An Incident Detection and Response managed EventBridge rule requires four specific fields, including source, detail-type, detail and incident-detection-response-identifier. The components in the architecture specify where these fields are implemented. The data flow is unidirectional and follows four simple blocks.

The different components in this architecture are the following:

• New Relic workflow: This is where you create the workflow and critical alerts in the New Relic account. Then, you create the incident-detection-response-identifier that’s required for Incident Detection and Response and add it to the payload. The article covers the workflow in detail in the configuration section.

• Partner event bus: New Relic integrates directly with the AWS account using EventBridge partner event bus.

• Lambda: This function is used for payload transformation where you provide the APM’s detail-type and source.

• EventBridge: You can set up a custom or default event bus to route the events. Incident Detection and Response requires the details of this bus to forward the critical alerts to the support team.

• Managed rule: Incident Detection and Response installs this managed rule.

• Incident Detection and Response: Incident Detection and Response creates the support cases automatically with data from the critical alert that New Relic raised.

Prerequisites

Make sure that you have the following:

• An AWS account with provisioned access for alert ingestion to Incident Detection and Response (Enterprise or Unified Operations Support plans) that’s already onboarded to Incident Detection and Response

• Permissions to create events in EventBridge and AWS Identity and Access Management (IAM) roles

• A New Relic account

• Permissions to create and manage New Relic workflows and alerts

• Permissions to create a Lambda function to transform the payload

• Access for alert ingestion to Incident Detection and Response

Solution implementation

Step 1: Configure the New Relic workflow

Create the workflow and a critical alert in the New Relic account. To configure the integration between New Relic and the AWS account, add the source details in the New relic account and destination details from AWS, such as the AWS account number and Region.

1. Sign in to your New Relic account.

2. Choose Alerts and then choose Workflows.

3. Choose Create your first workflow.

4. Enter a name for your workflow. Example: IDRIntegration

5. Under Filter Data, choose Advanced.

6. For Priority, select CRITICAL.
   This setting filters for alerts that are tagged as critical in nature.

7. Under Notify, for Add channel, choose Amazon EventBridge.
   You see a new widget to edit the notification message.

8. Under Edit notification message, in the Destination dropdown list, select Create new destination.
   You can see a widget to add a destination.

9. Enter a name for the new destination. Example: NewRelicIntegration

10. Choose Add a destination.

11. For Name, enter a name for the new destination. Example: NewRelicIDR

12. For AWS region, select your desired option.

13. For AWS account ID, enter the AWS account ID that you want to integrate.

14. Choose Save destination.
    You return to the Edit notification message widget.

15. Under Edit notification message, for Event source, enter a name for your event source. Example: NewRelicSource

16. Choose Done.

Step 2: Configure EventBridge

This section provides information about how to configure the partner event bus integration and associate the bridge between New Relic and AWS. As part of this configuration, you must add the unique identifier incident-detection-response-identifier to the New Relic workflow payload. The value for this identifier is the name of the alarm that you created within the workflow. This identifier helps Incident Detection and Response run the associated runbook as a response. To test if you configured the setup appropriately, complete the following steps:

1. Sign in to the AWS account that you want to integrate.

2. In the AWS Management console, make sure that the Region Selector shows the Region that you specified in the above procedure.

3. Open the Amazon EventBridge console.

4. In the navigation pane, choose Partner event sources.
   The event source that you created in the New Relic account appears in the list of partner event sources.

5. Select the event and then choose Associate with event bus.
   The status of the event source changes from Pending to Active, and the name of the event bus updates to match the partner event source name. You can now start creating rules that match events from the partner event source.

6. In the navigation pane, chose Event buses.

7. Select the partner event bus that you created and then choose Start discovery. This initiates the event bus listener for alerts from New Relic.

8. In the navigation pane, choose Rules, and then choose Create rule.

9. Enter a name and description for the rule.

10. For Event bus, select the partner event bus NewRelicSource that you created.

11. Choose Next.

12. In the Build event pattern page. for Event source, choose AWS events or EventBridge partner events.

13. Under Event Pattern, for Creation method, choose Use pattern form.

14. Do the following:
    For Event source, select EventBridge partners.
    For Partner, choose New Relic.
    For Event type, choose All Events.

15. Choose Next.

16. In the Select target(s) page, under Target 1, for Target types, select AWS service.

17. For Select a target, select CloudWatch log group.

18. For Log Group, select New log group. Then, enter a name for the new log group.

19. Choose Next.
    You use this CloudWatch log group to test the integration. When New Relic initiates a test notification, the logs capture the data that flows in.

20. (Optional) Choose Add new tag to add a new tag. Enter the tag key and value.

21. Choose Next.

22. In the Review and create page, review all the details you provided and then choose Create rule.
    You successfully created the rule.

Complete the following steps to add the incident and detection response identifier to your New Relic account so that the Incident and Response team can monitor the alerts:

1. Return to the New Relic console where you see the Edit notification message widget.

2. Under Edit notification message, for Configure event template, enter the following payload template:
   "workflowName": json workflowName add a new entry
   "incident-detection-response-identifier": {{json workflowName}}

The output looks like the following:

3. Choose Update message and then stay on this page.

Step 3: Test the integration

The next step is to test whether the critical alerts that New Relic generates reach AWS through the event bus that you configured in the previous section. To test the integration, you can send a test notification from New Relic that appears in the CloudWatch log group that you created.

Complete the following steps in the New Relic console:

1. In the New Relic console, under Edit notification message, choose Send test notification.

Complete the following steps in the CloudWatch console:

1. Open the CloudWatch console.

2. Choose Logs and then choose Log Management.

3. Choose the Log groups tab.

4. Choose the log group that you previously created.

5. Choose the Log streams tab.

6. Choose the message that CloudWatch most recently created.

7. Choose the down arrow to expand the message.

The message body contains details about the critical incident along with the custom field that you added to the template. The Incident Detection and Response team uses this data along with the context of the alert to achieve faster resolution times.

Step 4: Configure Incident Detection and Response integration

This section discusses how you can automatically redirect the events from New Relic to the Incident Detection and Response team for further investigation.

Incident Detection and Response requires the identifier, detail, detail-type, and source to automatically receive the events from New Relic. To set up the access that Incident Detection and Response requires for processing events from EventBridge, verify that a service-linked role exists. If it doesn’t exist, set up one. This information provides the required context for the alert, leading to faster resolution times.

Verify that the mandatory service-linked role AWSServiceRoleForHealth_EventProcessor exists. If it doesn’t exist, run the following command to create the role:

aws iam create-service-linked-role \\

--aws-service-name event-processor.health.amazonaws.com

You already set up the identifier incident-detection-response-identifier with the alarm name as part of the workflow payload transformation. The required values for Lambda transformation, such as detail-type and source, are specified in the EventBridge event bus. To learn how to add these values, see Create an AWS Lambda function for transformation. You can use the example transformation code template in Step 7. Be sure to replace the values to match the values that you set up for New Relic. You don’t have to specify the identifier because you already specified it in the workflow payload.

Cleanup

To avoid incurring costs for the following resources that you created in your AWS account, make sure to delete them:

• Amazon EventBridge custom event bus

• Amazon EventBridge rule

• New Relic workflow configurations

• Service-linked role

• Lambda function that you used for payload transformation

Conclusion

The New Relic integration with Incident Detection and Response helps with faster resolution times by providing detailed data about critical alerts. This integration also automates the transition of the alert for AWS support engineers to investigate without having to manually collect, prepare and send context and information about the alert. For more information on AWS Support plans, see AWS Support Plans. You can also reach out to your Solutions Architect or Technical Account Manager for further assistance.

About the author

Shafreen is a Specialist Solutions Architect with over 14 years of experience in designing, building, and implementing technical architectural patterns and best practices that drive security, resilience and operational excellence. She specialises in building scalable enterprise architectures that enhance user experience and business value. Passionate about data-driven solutions, Shafreen applies rigorous research and industry standards to bring meaning and context to complex information systems.