How to Forward Application Logs from AWS to Any SIEM Platform?

Overview

This solution provides a scalable, cost-effective, and SIEM-agnostic approach to forward application logs from AWS services to SIEM platforms for centralized logging and monitoring. While many SIEM platforms offer direct API ingestion capabilities, this S3-based architecture provides vendor independence and enhanced data durability for long-term retention and forensic analysis.

SIEM Agnostic Design: Compatible with any SIEM platform that supports S3 data sources (Splunk, QRadar, Sentinel, Elastic, etc.) with no vendor lock-in or proprietary API dependencies, ensuring consistent data format regardless of SIEM choice.

Dual Storage Strategy: S3 serves as long-term durable archival with lifecycle policies for cost optimization, while SIEM platforms handle short-term analytics with real-time analysis and alerting for operational insights. Historical data remains accessible in S3 even after SIEM retention expires.

Alternative Approach: Direct API integration offers real-time streaming with lower latency but requires SIEM-specific implementation and vendor lock-in. This S3-based solution is ideal if you need SIEM vendor flexibility, long-term data retention beyond your SIEM's storage limits, or want to future-proof your log architecture against vendor changes.

S3 as Final Destination: This solution delivers logs to Amazon S3 as the endpoint, providing a universal integration point that most enterprise SIEM platforms can consume through native S3 connectors. The actual S3-to-SIEM connection depends on your specific SIEM vendor's configuration capabilities, allowing you to leverage existing SIEM S3 integration features rather than requiring custom development.

High-Level Process

Application → CloudWatch Logs → Data Firehose → S3 → SIEM

1. Log Generation: Applications write logs to Amazon CloudWatch Logs
2. Stream Processing: CloudWatch Logs subscription filter triggers Amazon Data Firehose
3. Durable Storage: Firehose delivers logs to Amazon S3 with configurable batching and compression
4. SIEM Integration: SIEM platforms consume logs from S3 bucket for real-time analysis
5. Long-term Archival: S3 lifecycle policies manage cost-effective storage tiers for compliance and forensic investigation

Technical Implementation

Core Components Setup

1. CloudWatch Logs Configuration

• Configure log groups for application logs
• Set up subscription filters to trigger Firehose delivery stream

2. Amazon Data Firehose Setup

• Create delivery stream with S3 destination
• Configure buffering, compression, and error handling
• Reference: Stream CloudWatch Logs to Amazon S3 using Kinesis Data Firehose

3. Amazon S3 Configuration

• Create S3 bucket with appropriate folder structure
• Configure lifecycle policies for cost optimization
• Set up bucket notifications (optional) for real-time SIEM ingestion

4. SIEM Integration

• Configure SIEM S3 input to consume log files (Splunk, QRadar, Sentinel, etc.)
• Set up appropriate indexing and parsing rules
• Configure monitoring for S3 bucket changes

IAM Permissions Required

• CloudWatch Logs service role for Firehose access
• Firehose service role for S3 write permissions
• SIEM IAM user/role for S3 read access

Cost Components

AWS Service Costs (ap-southeast-3 region)

Amazon CloudWatch Logs:

• Log ingestion: $0.70 per GB (Custom logs, Standard class)
• Log storage: $0.03 per GB per month

Amazon Data Firehose:

• Data ingestion: $0.035 per GB

Amazon S3:

• Standard storage: $0.025 per GB per month
• PUT requests: $0.0005 per 1,000 requests
• GET requests: $0.0004 per 1,000 requests

Cost Estimation

Scenario: 1GB application logs per day (30GB/month)

Service	Component	Monthly Cost
CloudWatch Logs	Ingestion (30GB)	$21.00
CloudWatch Logs	Storage (30GB)	$0.90
Data Firehose	Processing (30GB)	$1.05
Amazon S3	Storage (30GB)	$0.75
Amazon S3	API Requests	$0.03
Total		$23.73

Benefits

Operational: The SIEM-agnostic architecture supports vendor flexibility and migration while providing a decoupled design that enables independent scaling of storage and analytics. Serverless components reduce operational overhead, and S3 provides 99.999999999% (11 9's) durability for critical log data.

Cost Optimization: S3 lifecycle policies automatically transition data to cheaper storage classes while Firehose batching reduces S3 API costs. The solution requires no infrastructure management or maintenance and implements a dual retention strategy with SIEM for short-term analysis and S3 for long-term storage.

Forensic and Compliance: Long-term log retention in S3 extends beyond SIEM storage limits, providing historical analysis capabilities for incident investigation. The architecture creates an immutable audit trail with S3 versioning and object lock while ensuring cost-effective compliance with data retention regulations.

Considerations

• Latency: Near real-time delivery (30 seconds to 5 minutes depending on Firehose buffering)
• Scalability: Automatically scales with log volume
• Monitoring: Implement CloudWatch metrics for delivery stream health
• Security: Enable S3 encryption and secure SIEM access patterns

Try the Solution

You can deploy and test this solution using the provided Infrastructure as Code (IaC) template [Appendix A - deploy-cloudformation.json]. The deployment creates a complete working example with a Lambda function that generates test logs.

Prerequisites

• AWS CLI configured with appropriate permissions
• CloudFormation deployment permissions
• Access to ap-southeast-3 (Jakarta) region

Quick Deployment

Option 1: AWS CLI Deployment

# Generate random suffix and deploy
RANDOM_SUFFIX=$(date +%s | tail -c 6)
STACK_NAME="cw-logs-test-${RANDOM_SUFFIX}"

aws cloudformation deploy \
  --template-file deploy-cloudformation.json \
  --stack-name "$STACK_NAME" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides RandomSuffix="$RANDOM_SUFFIX" \
  --region ap-southeast-3

Option 2: AWS Console Deployment

1. Download the deploy-cloudformation.json template
2. Go to CloudFormation console in ap-southeast-3 region
3. Create stack → Upload template file
4. Set stack name and RandomSuffix parameter
5. Deploy with IAM capabilities

Testing the Solution

# Test log generation
aws lambda invoke \
  --function-name test-${RANDOM_SUFFIX}-function \
  --region ap-southeast-3 \
  response.json

# Check S3 logs (wait 1-2 minutes)
aws s3 ls s3://test-${RANDOM_SUFFIX}-logs-${ACCOUNT_ID}-ap-southeast-3/logs/ --recursive

What Gets Deployed

• Lambda Function: Generates test application logs
• CloudWatch Log Group: Collects Lambda logs
• Kinesis Data Firehose: Streams logs to S3
• S3 Bucket: Stores log files for SIEM consumption
• IAM Roles: Required permissions for all components
• Subscription Filter: Connects CloudWatch Logs to Firehose

Expected Costs

Running this test for a few hours will generate minimal costs (< $0.10) and demonstrates all the usage types discussed in the cost analysis.

Cleanup

# Empty S3 bucket and delete stack
aws s3 rm s3://test-${RANDOM_SUFFIX}-logs-${ACCOUNT_ID}-ap-southeast-3 --recursive
aws cloudformation delete-stack --stack-name "$STACK_NAME" --region ap-southeast-3

Appendix A - deploy-cloudformation.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Minimal test infrastructure for CloudWatch Logs to Firehose to S3",
  "Parameters": {
    "RandomSuffix": {
      "Type": "String",
      "Default": "123456",
      "Description": "Random suffix for unique resource naming"
    }
  },
  "Resources": {
    "LogsBucket": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "BucketName": {
          "Fn::Sub": "test-${RandomSuffix}-logs-${AWS::AccountId}-${AWS::Region}"
        },
        "PublicAccessBlockConfiguration": {
          "BlockPublicAcls": true,
          "BlockPublicPolicy": true,
          "IgnorePublicAcls": true,
          "RestrictPublicBuckets": true
        }
      }
    },
    "FirehoseRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {
          "Fn::Sub": "test-${RandomSuffix}-firehose-role"
        },
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "firehose.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "Policies": [
          {
            "PolicyName": "FirehoseS3Policy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:AbortMultipartUpload",
                    "s3:GetBucketLocation",
                    "s3:GetObject",
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads",
                    "s3:PutObject"
                  ],
                  "Resource": [
                    {
                      "Fn::GetAtt": ["LogsBucket", "Arn"]
                    },
                    {
                      "Fn::Sub": "${LogsBucket.Arn}/*"
                    }
                  ]
                }
              ]
            }
          }
        ]
      }
    },
    "LogsFirehose": {
      "Type": "AWS::KinesisFirehose::DeliveryStream",
      "Properties": {
        "DeliveryStreamName": {
          "Fn::Sub": "test-${RandomSuffix}-logs-stream"
        },
        "DeliveryStreamType": "DirectPut",
        "S3DestinationConfiguration": {
          "BucketARN": {
            "Fn::GetAtt": ["LogsBucket", "Arn"]
          },
          "Prefix": "logs/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/",
          "ErrorOutputPrefix": "errors/",
          "BufferingHints": {
            "SizeInMBs": 5,
            "IntervalInSeconds": 30
          },
          "CompressionFormat": "GZIP",
          "RoleARN": {
            "Fn::GetAtt": ["FirehoseRole", "Arn"]
          }
        }
      }
    },
    "TestLogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": {
          "Fn::Sub": "/aws/lambda/test-${RandomSuffix}-function"
        },
        "RetentionInDays": 7
      }
    },
    "LogsRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {
          "Fn::Sub": "test-${RandomSuffix}-logs-role"
        },
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": {
                  "Fn::Sub": "logs.${AWS::Region}.amazonaws.com"
                }
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "Policies": [
          {
            "PolicyName": "LogsFirehosePolicy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "firehose:PutRecord",
                    "firehose:PutRecordBatch"
                  ],
                  "Resource": {
                    "Fn::GetAtt": ["LogsFirehose", "Arn"]
                  }
                }
              ]
            }
          }
        ]
      }
    },
    "LogsSubscriptionFilter": {
      "Type": "AWS::Logs::SubscriptionFilter",
      "Properties": {
        "LogGroupName": {
          "Ref": "TestLogGroup"
        },
        "FilterPattern": "",
        "DestinationArn": {
          "Fn::GetAtt": ["LogsFirehose", "Arn"]
        },
        "RoleArn": {
          "Fn::GetAtt": ["LogsRole", "Arn"]
        }
      }
    },
    "LambdaRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {
          "Fn::Sub": "test-${RandomSuffix}-lambda-role"
        },
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "lambda.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "ManagedPolicyArns": [
          "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        ]
      }
    },
    "TestLambda": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "FunctionName": {
          "Fn::Sub": "test-${RandomSuffix}-function"
        },
        "Runtime": "python3.9",
        "Handler": "index.lambda_handler",
        "Role": {
          "Fn::GetAtt": ["LambdaRole", "Arn"]
        },
        "Code": {
          "ZipFile": "import json\nimport logging\nimport random\nimport time\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n\ndef lambda_handler(event, context):\n    for i in range(10):\n        logger.info(f'Test log entry {i+1}: Processing request with ID {random.randint(1000, 9999)}')\n        logger.warning(f'Warning message {i+1}: High memory usage detected')\n        time.sleep(0.1)\n    \n    logger.error('Test error message: Database connection timeout')\n    \n    return {\n        'statusCode': 200,\n        'body': json.dumps('Test logs generated successfully')\n    }"
        }
      }
    }
  },
  "Outputs": {
    "S3BucketName": {
      "Description": "S3 bucket for storing logs",
      "Value": {
        "Ref": "LogsBucket"
      }
    },
    "FirehoseStreamName": {
      "Description": "Firehose delivery stream name",
      "Value": {
        "Ref": "LogsFirehose"
      }
    },
    "LogGroupName": {
      "Description": "CloudWatch Log Group name",
      "Value": {
        "Ref": "TestLogGroup"
      }
    },
    "LambdaFunctionName": {
      "Description": "Lambda function name for testing",
      "Value": {
        "Ref": "TestLambda"
      }
    }
  }
}