AWS Incident Detection and Response: Enhancing critical workload monitoring with Splunk Observability Cloud integration

Introduction

In today's cloud-first world, organizations need robust incident detection and response capabilities to maintain high availability and reliability. AWS Incident Detection and Response provides 24/7 monitoring, detection, and response capabilities for critical workloads that run on AWS. The service combines the operational expertise of AWS with advanced tooling to help customers detect and respond to operational events quickly. For organizations that run mission-critical workloads, integrating AWS Incident Detection and Response with existing application performance monitoring (APM) tools, such as Splunk Observability Cloud, creates a unified monitoring and response ecosystem. This integration empowers Splunk's critical observability alarms to flow seamlessly into the AWS Incident Detection and Response framework. Through this integration, organizations can achieve faster incident resolution and reduced mean time to recovery (MTTR).

AWS Incident Detection and Response and Splunk Observability Cloud integration architecture

The integration between AWS Incident Detection and Response and Splunk Observability Cloud uses Amazon EventBridge and AWS Lambda to process and route monitoring alerts. You can configure Splunk Observability Cloud to automatically send alert notifications to EventBridge when a detector alert condition is met, as seen in Figure 1.

Figure 1: High-level integration overview.

The architecture consists of several key components:

1. Splunk Observability Cloud: Monitors your applications and infrastructure and generates alerts based on your defined thresholds and conditions.
2. Partner event source: A connection in EventBridge that allows Splunk Observability Cloud to send events to your AWS account.
3. EventBridge: Serves as the event bus that receives alerts from Splunk Observability Cloud through the partner event source.
4. EventBridge rules: Routes relevant alerts to the AWS Incident Detection and Response service based on predefined patterns.
5. AWS Incident Detection and Response: Receives the alerts and initiates the appropriate response procedures according to your engagement model.
6. Incident response actions: The specific actions that the AWS Incident Detection and Response team takes to address the incident.

Integrating AWS Incident Detection and Response and Splunk Observability Cloud

Configure Splunk Observability Cloud to send alerts to EventBridge

To create an EventBridge integration in Splunk Observability Cloud, you must have Splunk Observability Cloud administrator permissions. To create the integration, complete the following steps:

1. As an administrator, open the Splunk Observability Cloud.
2. Choose Data Management, and then choose Available Integrations.
3. In the Search field, search for and select Amazon EventBridge, as seen in Figure 2.

Figure 2: EventBridge integration.

4. On the EventBridge page, choose Add Integration.
5. In the Create integration dialog box, add the details of the AWS Account ID and AWS Region where you set up the integration, as seen in Figure 3.

Figure 3: Create integration dialog box details.

Note: The Event Source field displays the name of the partner event source. This partner event source is available in EventBridge in the selected account and Region after you complete the setup.

6. Choose Save and enable. The EventBridge integration shows an Active status.
7. Configure the existing Splunk Observability Cloud detectors to send the alert notifications to the EventBridge integration that you configured. Choose the detectors that correspond to your critical application alerts. Then, under Alert recipients, choose Add recipient, as seen in Figure 4.

Figure 4: Add recipient to Alert recipient.

8. Choose Amazon EventBridge, and then choose Proceed To Alert Activation, as seen in Figure 5.

Figure 5: Proceed to Alert Activation.

9. Under Activate, choose Update Alert Rule, as seen in Figure 6.

Figure 6: Update Alert Rule.

Configure EventBridge to receive alert notifications from Splunk Observability Cloud

After you configure Splunk Observability Cloud, configure EventBridge to receive alert notifications:

1. Open the AWS Management Console in the account and Region where you created the EventBridge integration on the Splunk Observability Cloud.
2. Open the EventBridge console.
3. In the navigation pane, under Integration, choose Partner event resources. Under Partner event sources, you can see the Splunk Observability Cloud integration that you configured with a Status of Pending.
4. To associate the Partner event source with a Partner event bus, select the event source and choose Associate with event bus, as seen in Figure 7.

Figure 7: Associate the Partner event source with a Partner event bus.

5. In the Associate with event bus page, choose Associate.

You can see that the Splunk Observability Cloud event source aws.partner/signalfx.com/ is Active. EventBridge also creates an associated custom event bus with the same name as the event source aws.partner/signalfx.com/. The event bus processes the events from the event source, as seen in Figure 8.

Figure 8: Partner event source and event bus.

Create a custom EventBridge event bus to ingest transformed events

AWS Incident Detection and Response uses a custom event bus to ingest transformed events received from Splunk Observability Cloud. Create a custom event bus to ingest the transformed events from the Splunk Observability Cloud event bus. To do this, complete the following steps:

1. Open the EventBridge console.
2. In the navigation pane, under Buses, choose Event buses. Under Custom event bus, choose Create. Then, provide a name and description for your event bus. Enter the name in a format similar to the following example, Splunk-AWSIncidentDetectionResponse-EventBus, as seen in Figure 9.

Figure 9: Create an event bus.

3. Choose Create. You can find the event bus that you created on the Event buses page, as seen in Figure 10.

Figure 10: Event buses page.

Create a Lambda function

Create a Lambda function to transform the events from the Splunk Observability Cloud event bus. The Lambda function can send the events to a custom event bus that AWS Incident Detection and Response uses. Complete the following steps:

1. Open the Lambda console.
2. In the navigation pane, under Functions, choose Create function.
3. On the Create function page, choose Author from scratch.
   For Function name, enter a name similar to the following example, Splunk-AWSIncidentDetectionResponse-LambdaFunction.
   For Runtime, choose Python 3.13.
4. Choose Create function, as seen in Figure 11.

Figure 11: Create a Lambda function.

5. Configure a Lambda function to transform events between the partner event bus aws.partner/signalfx.com/ and the custom event bus Splunk-AWSIncidentDetectionResponse-EventBus that you created. AWS Incident Detection and Response requires the following JSON key:value pairs in the alert payload. The Lambda function transforms the alert to include the required JSON key:value pairs:

{
    "detail-type": "ams.monitoring/generic-apm",
    "source": "GenericAPMEvent",
    "detail" : {
       "incident-detection-response-identifier": "example-alarm-name"
    }
}

6. On the Code Edit page, replace the default Lambda function content with the following code example:

import logging
import json
import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

EventBusName = "Splunk-AWSIncidentDetectionResponse-EventBus"

def lambda_handler(event, context):
   event["detail"][" incident-detection-response-identifier"] = event["detail"]["ruleName"]
    logger.info(f"We got: {json.dumps(event, indent=2)}")   

    client = boto3.client('events')
    response = client.put_events(
        Entries=[
            {
                'Detail': json.dumps(event["detail"], indent=2),
                'DetailType': 'ams.monitoring/generic-apm',  # Do not modify. This DetailType value is required.
                'Source': 'GenericAPMEvent',  # Do not modify. This Source value is required.
               'EventBusName': EventBusName  # Do not modify. This variable is set at the top of this code as a global variable. Change the variable value for your eventbus name at the top of this code.
            }
        ]
    )
    print(response['Entries'])

Note: Replace EventBusName with the custom event bus that you created for AWS Incident Detection and Response. Replace example-alarm-name with the name of the alarm for your APM.

7. Choose Deploy, as seen in Figure 12.

Figure 12: Code edit page.

8. To configure the Lambda function to send events to the custom event bus, provide PutEvents permissions to the Lambda execution role. In the Lambda console, under Functions, select the function Splunk-AWSIncidentDetectionResponse-LambdaFunction.
9. Under Configuration, choose Permissions, and then select the Role Name. The example role name is Splunk-AWSIncidentDetectionResponse-LambdaFunction-role-78m5qkbi, as seen in Figure 13.

Figure 13: Edit the execution role.

10. When you select the Role name, the AWS Identity and Access Management (IAM) console opens to edit the execution role, as seen in Figure 14. Under Permissions, choose Permissions Policies, and then select the existing policy name to edit the policy.

Figure 14: Edit the permissions for the execution role.

11. On the Policy editor page, select Add new statement, as seen in Figure 15.

Figure 15: Add a statement to the policy.

12. The Policy editor adds a new blank statement. Replace the auto-generated statement with the following information:
    For Resource, enter the Amazon Resource Name (ARN) of the custom event bus that you created.

 {
"Sid": "AWSIncidentDetectionResponseEventBus0",
  "Effect": "Allow",
  "Action": "events:PutEvents",
  "Resource": "arn:aws:events:{region}:{accountId}:event-bus/Splunk-AWSIncidentDetectionResponse-EventBus"
}

13. Confirm that you added the required permission to the role.
14. Choose Set this new version as the default, and then choose Save changes, as seen in Figure 16.

Figure 16: Save your IAM role.

Create a custom EventBridge rule

Create a custom EventBridge rule to send the events from Splunk Observability Cloud event bus to the Lambda function for transformation:

1. Open the EventBridge console.
2. In the navigation pane, choose Event buses, and then select the aws.partner/signalfx.com/ event bus, as seen in Figure 17.

Figure 17: Select the event bus.

3. Navigate to Rules, select Create rule.
4. For rule name, enter a name in the following format: APMName-AWS Incident Detection and Response-EventBridgeRule. Then, choose Next.
5. For Event source, select AWS events or EventBridge partner events and enter the following information:
   For Creation method, choose Use pattern form.
   For Event source, choose EventBridge partner.
   For Partner, select your APM Partner.
   For Event Type, choose All events, and then choose Next, as seen in Figure 18.

Figure 18: Configure the event pattern.

6. For Targets, enter the following information:
   For Target types, choose AWS service.
   For Select a target, choose Lambda function.
   For Function, choose Splunk-AWSIncidentDetectionResponse-LambdaFunction, as seen in Figure 19.

Figure 19: Select your targets.

7. Choose Next, and then choose Next.
8. Choose Save Rule.

Conclusion

When you integrate AWS Incident Detection and Response with Splunk Observability Cloud, you create a proactive monitoring and response system for your critical workloads on AWS. This integration delivers rapid response capabilities so that your teams can avoid incidents and reduce the MTTR for incidents when they occur. AWS Incident Management Engineers monitor the alarms that Splunk creates 24/7. These engineers respond within 5 minutes of critical incidents through the custom response plans developed beforehand during AWS Incident Detection and Response onboarding.

To get started with AWS Incident Detection and Response, contact your AWS Account Manager. Or, see AWS Incident Detection and Response | Contact us.

About the author

Chandra Allaka

Chandra is a principal specialist solutions architect with AWS Enterprise Support. He’s passionate about helping customers with their cloud operations excellence and building effective solutions to manage cloud environments at scale.