Improving automated response time: Integrating AWS Incident Detection and Response with Sumo Logic

Introduction

Organizations that implement automated incident response procedures regularly face the following challenges:

• Alert fatigue from multiple monitoring tools.
• A lack of unified visibility across their AWS infrastructure.
• Manual remediation processes that introduce dangerous delays between threat detection and response.

Traditional operations centers receive thousands of alerts daily from multiple sources: Amazon CloudWatch, Amazon GuardDuty, AWS Security Hub, and third-party SIEM platforms such as Sumo Logic. Without intelligent correlation and automated workflows, security analysts spend valuable time triaging false positives, manually investigating incidents, and executing repetitive remediation tasks that could be automated. This reactive approach creates security gaps where undetected threats can persist, compliance violations go unaddressed, and mean time to resolution (MTTR) extends from hours to days. These gaps expose your organization to increased risk and potential effects.

To resolve these issues, you can integrate AWS Incident Detection and Response with Sumo Logic's advanced analytics platform to create an intelligent, automated security operations workflow. This workflow can detect threats in real-time, correlate events across your entire infrastructure, prioritize alerts based on business context, and automatically run-defined remediation playbooks. This solution transforms security operations from reactive firefighting to proactive threat prevention and reduces MTTR by up to 90%.

Solution overview

The Sumo Logic and AWS Incident Detection and Response solution creates an automated workflow that enhances incident detection and response in AWS environments. This integration combines Sumo Logic’s log analysis capabilities with AWS Incident Detection and Response’s incident management features to streamline log collection, anomaly detection, alert generation, and incident response. The solution creates secure event processing, transforms events into a compatible format, automatically creates incidents with relevant log context, and notifies response teams with detailed information. It also improves proactive incident detection, automates workflows, enhances security, and allows for faster, more informed incident responses, leading to more resilient and reliable AWS environments.

Architectural flow

Figure 1: The incident detection workflow.

The incident detection workflow seen in Figure 1 begins when Sumo Logic's Open Telemetry Collector gathers logs from AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and AWS Lambda. Sumo Logic monitors use anomaly detection algorithms to analyze these logs in real-time to identify error patterns. When the solution detects critical conditions, webhook notifications trigger Amazon API Gateway to authenticate the requests before passing them to Lambda functions. These Lambda functions transform Sumo Logic events into AWS Incident Detection and Response-compatible formats. AWS Incident Detection and Response then automatically creates incidents that contain log context and metadata, such as timestamps and severity levels. Finally, incident response teams receive detailed notifications through channels, such as email or Slack, so that they can follow predefined runbook steps to address the incidents.

Solution implementation

To implement the solution, complete the following steps.

Deploy the CloudFormation stack

Deploy the CloudFormation stack to receive webhooks from Sumo Logic, and then forward the webhooks to AWS Incident Detection and Response. To deploy the webhook configuration, complete the steps on Third party application performance monitoring (APM) setup on the GitHub website. When you configure the APM setup, for Provider, select Dynatrace. Sumo Logic follows the same steps as this provider option.

After you deploy the CloudFormation stack, retrieve the webhook URL and authentication token for Sumo Logic configuration from the CloudFormation stack outputs tab. 

Create a Sumo Logic workflow

For the solution to collect log information, you must create a Sumo Logic workflow. To create a Sumo Logic workflow, use Session Manager, a capability of AWS Systems Manager, to connect to your Amazon EC2 instance. Then, run the following command to start your Systems Manager session and start collecting logs:

aws ssm start-session --target YOUR_INSTANCE_ID

Note: Replace YOUR_INSTANCE_ID with your instance ID information from CloudFormation.

Or, use the Amazon EC2 console to connect to your EC2 instance and access the Session Manager.

Configure the Sumo Logic collector
To configure the Sumo Logic collector, complete the following steps:

1. Open the Sumo Logic console, and then open the App Catalog.
2. Choose Linux, and then choose Linux OpenTelemetry. 
3. Follow the configuration wizard to set up the collector for your environment.
4. Copy the installation commands.
5. Use the Systems Manager session to run the commands in your EC2 instance.

Install the Sumo Logic collector
To install the Sumo Logic collector, run the installation commands:

# Example commands  
sudo curl -sL https://download-otel.sumologic.com/latest/download/install.sh | \
  SUMOLOGIC_INSTALLATION_TOKEN="YOUR_TOKEN_HERE" sudo -E bash -s -- \
  --tag "host.group=default" --tag "deployment.environment=default"

Note: Use the installation commands that are in the Sumo Logic console.

Review the installation
To verify that you correctly installed the collector, run the following command to check the status of the collector:

sudo systemctl status otelcol-sumo

The status should return an Active (running) status. If the status isn’t correct, then see Troubleshooting on the Sumo Logic website.

To review your collector logs, run the following command:

# View collector logs  
sudo journalctl -u otelcol-sumo -f

To review your data, open the Sumo Logic console. Choose Manage, and then choose Collection. The collector status appears as Online. To review specific log data, use the Search field and filter your log data by _soureHost=*.

Create a webhook connection

To create a webhook connection, complete the following tasks.

Access your webhook URL
To access your webhook URL, complete the following steps:

1. Open the Sumo Logic console.
2. Under Monitoring, choose Connections, as seen in Figure 2.

Figure 2: Connections in the Sumo Logic console.

3. Choose Webhook, and then enter the webhook URL from your CloudFormation stack outputs, as seen in Figure 3.

Figure 3: Select a webhook connection type.

Get the authentication token from Secrets Manager
To allow access to the webhook, run the following command in Systems Manager to get the authentication token from AWS Secrets Manager:

aws secretsmanager get-secret-value \
  --secret-id "SumoLogicMySecretTokenName" \
  --query 'SecretString' \
  --output text \| jq -r '.APMSecureToken'

Note: Replace SumoLogicMySecretTokenName and SecretString with your information.

Or, use the Secrets Manager console to find your authentication token. Choose Retrieve secret value, and then copy the APMSecureToken value from the JSON.

Configure your webhook settings
Configure the webhook connection with the URL and authentication token, as seen in Figure 4.

Webhook connection example:

{  
  "url": "https://{api-id}.execute-api.{region}.amazonaws.com/{stage}/APIGWResourcesforAPM",  
  "method": "POST",  
  "headers": {  
    "Content-Type": "application/json",  
    "authorizationToken": "{token-from-secrets-manager}"  
  }  
}

Figure 4: Configure the webhook with your URL and authentication token.

Create an alert payload template
For AWS Incident Detection and Response to successfully process the payload, use the following example to create a payload template:

{  
  "detail": {  
    "ProblemTitle": "{{AlertName}}",  
    "State": "{{TriggerType}}",  
    "ProblemID": "{{AlertId}}",  
    "ImpactedEntity": "{{Query}}",  
    "Severity": "{{TriggerValue}}"  
  }  
}

Note: Include the detail object.

Create log monitors

To detect error patterns in your logs and initiate webhook notifications to AWS Incident Detection and Response, you can create monitors in Sumo Logic. To do so, complete the following steps:

1. In the Sumo Logic console, under Monitoring, choose Monitors, as seen in Figure 5.

Figure 5: Choose Monitors in the Sumo Logic console.

2. Choose Add monitor, and then select Logs for the monitor type.
3. Configure your query and alert conditions to specify what error patterns trigger notifications. Under Notifications, select your AWS Incident Detection and Response webhook connection and implement the payload template that you created.

Sumo Logic automatically sends properly formatted webhook notifications to AWS Incident Detection and Response that meet your conditions. To verify that the notifications work, use the Sumo Logic console to send a test alert. If you successfully configured the test alert, then it returns a 200 response code, as seen in Figure 6. 

Figure 6: Successful 200 code returned.

To monitor for high error rate in application logs, run the following query:

_sourceCategory=application/logs error | timeslice 5m | count by _timeslice | where _count > 10

Clean up

To delete the deployed resources and make sure that you don’t continue to incur charges, run the following command:

aws cloudformation delete-stack --stack-name your-stack-name

Note: You can also delete the stack from the CloudFormation console.

Conclusion

When organizations integrate AWS Incident Detection and Response with Sumo Logic monitoring capabilities, they streamline the incident response process. This integration reduces the MTTR through the following benefits:

• Automates the collection and analysis of logs, minimizing the need for manual intervention.
• Makes sure that incident response teams have all the necessary log context and details.
• Preserves current workflows while incorporating advanced automation to improve efficiency and effectiveness.

To learn more about how AWS Incident Detection and Response and Sumo Logic integration can help you get the most out of your AWS environment, see the AWS Incident Detection and Response documentation. For implementation support, contact Sumo Logic Support for workflow and monitoring configuration assistance. Or, contact AWS Support for AWS Incident Detection and Response configuration guidance.

About the authors

Kisshore Gunasekaran
Kisshore Gunasekaran is a Senior Specialist Solutions Architect at AWS. He focuses on helping customers build secure cloud foundations and is passionate about solving complex operational and security challenges. Kisshore uses automation and best practices to accelerate cloud adoption. He works closely with enterprise customers and provides them with practical guidance to innovate and build secure scalable solutions on AWS.

Anjani Reddy
Anjani Reddy is a Senior Solutions Architect at AWS. She works with enterprise customers to provide operational guidance to innovate and build a secure, scalable environment in the AWS Cloud.