How to harmonize logs in Control Tower Logging Account

Hello Andre,

I think AWS Control Tower has certain limitations when it comes to changing the default log storage structure for AWS services like CloudTrail and Config.

However, you can work around these limitations by configuring log data forwarding from Control Tower's logging account to your desired centralized location. Try the following high-level approach to harmonize logs from multiple AWS services in a multi-account setup:

1. Centralized Logging Account Setup:

   • Ensure that you have a dedicated centralized logging account that will receive and harmonize logs from multiple member accounts.

2. Create S3 Buckets for Harmonized Logs:

   • In the centralized logging account, create S3 buckets with the desired structure for each AWS service you want to aggregate logs for. For example:

     s3://central-logging-bucket/CloudTrail/
     s3://central-logging-bucket/Config/
     

3. Set Up Cross-Account Access:

   • In each member account, configure cross-account access permissions to allow the centralized logging account to access the S3 buckets created in step 2. Use bucket policies and IAM roles to establish these permissions.

4. Configure CloudTrail Log Forwarding:

   • In each member account, configure CloudTrail to deliver log files to the centralized logging S3 bucket created for CloudTrail logs. You can set up log file validation to ensure data integrity during transit.

5. Configure Config Log Forwarding (Optional):

   • Similar to CloudTrail, you can configure AWS Config to deliver log files to the centralized logging S3 bucket created for Config logs. This might require custom Lambda functions or other mechanisms depending on your specific requirements.

6. Other AWS Service Log Forwarding (Optional):

   • For other AWS services like VPC Flow Logs or Lambda logs, follow similar procedures to configure log forwarding to the centralized logging S3 buckets.

7. Harmonize Data in Centralized Storage:

   • Once logs are delivered to the centralized S3 buckets, you can implement Lambda functions, AWS Glue, or other ETL (Extract, Transform, Load) processes to harmonize the data into the desired structure. You can organize the data by account, service, and time as needed.

8. Set Up Log Analytics and Monitoring:

   • Implement log analytics and monitoring solutions in the centralized logging account to efficiently search, analyze, and visualize the log data. AWS services like Amazon Athena, Amazon Elasticsearch, or third-party tools can help with this.

9. Testing and Validation:

   • Ensure that log forwarding and harmonization processes work as expected by testing them in a controlled environment. Verify that logs are organized and stored according to your requirements.

10. Documentation and Maintenance:

    • Document the log forwarding and harmonization processes and ensure that they are well-documented for future reference and maintenance.

Please give a thumbs if my suggestion helps