How to Combine Split JSON Logs from AWS Elastic Beanstalk into a Single Log Entry Using CloudWatch Logs Processor?

Hi Esteban!

This is a great question, and it's a common challenge when dealing with logs in AWS Elastic Beanstalk. Let’s tackle it step by step so you can streamline your logs and make them more readable. 😊

---

Clarifying the Issue

Your environment is generating logs in the web.stdout.log file line by line, which causes JSON messages and stack traces to split into multiple lines. This fragmentation makes it difficult to capture all relevant information in a single log entry. You’ve already tried regex patterns and Lambda preprocessing, but consistency has been a problem. The goal is to consolidate these logs into cohesive entries in CloudWatch.

---

Key Terms

• AWS Elastic Beanstalk: A service for deploying and managing applications in the cloud.
• CloudWatch Logs: A service for monitoring, storing, and accessing log files from various AWS resources.
• Log Streams: Sequences of log events that share the same source.
• Subscription Filter: A mechanism to forward CloudWatch Logs to a specific destination, like a Lambda function.
• IAM Role: A set of permissions that a Lambda function or AWS service requires to perform its actions.
• ARN (Amazon Resource Name): A unique identifier for AWS resources. For example, a Lambda function ARN looks like arn:aws:lambda:region:account-id:function:function-name. This is used to reference resources across AWS services.

---

The Solution (Our Recipe)

Steps at a Glance

1. Enable Log Streaming: Configure Elastic Beanstalk to send logs to CloudWatch Logs.
2. Set Up a CloudWatch Logs Subscription: Forward logs to a Lambda function for processing.
3. Write a Lambda Function: Merge multiline log entries into cohesive JSON objects.
4. Stream Processed Logs: Configure the Lambda function to write the processed logs back to CloudWatch or another destination.

---

Detailed Steps

Step 1: Enable Log Streaming

In the Elastic Beanstalk console:

• Navigate to the Monitoring tab.
• Ensure that log streaming is enabled, and the logs are being sent to a specific CloudWatch Logs group (e.g., /aws/elasticbeanstalk/environment-name).

---

Step 2: Set Up a CloudWatch Logs Subscription

1. Open the CloudWatch Logs console and select the log group receiving your Beanstalk logs.
2. Create a subscription filter with your Lambda function's ARN:
   • Click Actions > Create Data Subscription.
   • Enter the ARN of the Lambda function that will process your logs.
3. Ensure the Lambda function has the AWSLambdaBasicExecutionRole policy and permissions for logs:PutLogEvents and logs:CreateLogStream.

---

Step 3: Write a Lambda Function

Create a Python-based Lambda function to parse and combine logs. Here’s an example:

import json

def lambda_handler(event, context):
    combined_logs = []
    partial_log = None
    
    for record in event['records']:
        message = record['message']
        if message.startswith('{'):
            if partial_log:
                combined_logs.append(partial_log)
            partial_log = message
        elif partial_log:
            partial_log += "\n" + message
            if message.endswith('}'):
                combined_logs.append(partial_log)
                partial_log = None

    # Return modified logs
    return {
        'records': [
            {'recordId': record['recordId'], 'result': 'Ok', 'data': json.dumps({'message': log})}
            for record, log in zip(event['records'], combined_logs)
        ]
    }

Sample Input (Before Processing):

17:34:09 INFO { "status": "success", "responseCode": 200
17:34:10 INFO "requestId": "abc123", "duration": "34ms" }
17:34:11 ERROR { "status": "failure", "responseCode": 500
17:34:12 ERROR "errorMessage": "Internal Server Error" }

Expected Output (After Processing):

[
    { "status": "success", "responseCode": 200, "requestId": "abc123", "duration": "34ms" },
    { "status": "failure", "responseCode": 500, "errorMessage": "Internal Server Error" }
]

---

Step 4: Stream Processed Logs

1. Modify the Lambda function to write processed logs back to CloudWatch or an S3 bucket:
   • Use the CloudWatch Logs API for writing to a new log group.
   • Alternatively, configure the function to store unified logs in S3 for long-term storage.
2. To write back to CloudWatch Logs, ensure the Lambda function’s IAM role includes logs:PutLogEvents and logs:CreateLogStream permissions.

---

Closing Thoughts

By following this solution, you’ll be able to combine fragmented logs into cohesive entries, improving readability and usability. Sending the processed logs back to CloudWatch ensures seamless integration with your existing workflows for monitoring and alerting. Let me know if you need further help with any of these steps! 🚀

---

Good luck, Esteban! Consolidating your logs will make troubleshooting and monitoring so much easier. Feel free to share updates or follow-up questions.

---

Cheers,

Aaron 😊