How do I troubleshoot CloudWatch logs that fail to export to S3 buckets?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you use the most recent AWS CLI version.

Troubleshoot based on whether the logs fail when you're creating the export task or after you create the export task.

Logs don't export during task creation

To troubleshoot tasks that fail when you create them, take the following actions:

• Confirm that your CloudWatch log streams and Amazon S3 buckets are in the same AWS Region.
• If you're not the S3 bucket owner, then check whether you have permission to access the bucket. Only the resource owner has access to the bucket and the objects in the bucket. For more information, see Set permissions on an Amazon S3 bucket (console).
• When you create the bucket policy, it's a best practice to include a randomly generated string as the prefix for the bucket. If you use a prefix, then you must specify the generated string in the bucket prefix settings when you create the export task. If you don't specify the generated string, then the export task creation fails.
• Confirm that the AWS Identity and Access Management (IAM) user or role that created the export task has access to Amazon S3 and CloudWatch Logs. For more information, see Set up access permissions (console).
• Check your AWS CloudTrail logs for the CreateExportTask event to determine the error that caused the task to fail. 
• Check whether you exceeded the CloudWatch Logs service quota. The maximum number is one running or pending export task for each AWS account in each Region. You can't change this quota.
• Check whether you're using a supported server-side encryption for your buckets. You can use either Amazon S3 managed keys (SSE-S3) or AWS Key Management Service (AWS KMS) keys (SSE-KMS). 

Logs don't export after task creation

To troubleshoot tasks that fail after you create them, check the time range setting. If you export log streams that have large amounts of data and specify a long time range, then the export task might fail. To resolve this issue, configure shorter time ranges to export log streams.

To check the status of the time range setting, run the following describe-export-tasks AWS CLI command:

aws logs describe-export-tasks --task-id example-task-id

Note: Replace example-task-id with your task ID.

To shorten the time range, run the following create-export-task AWS CLI command and modify the --from and --to values:

aws logs create-export-task --task-name "example-task-name" --log-group-name "/aws/lambda/example-log-group-name" --destination "example-bucket-name --from 1710378735 --to 1710465135

Note: Replace example-task-name with your task name, example-log-group-name with your log group name, example-bucket-name with your destination bucket.

To check the status of the export task, run the following describe-export-tasks AWS CLI command:

aws logs describe-export-tasks --task-id example-task-id

Note: Replace example-task-id with your task ID.

The logs might take up to 12 hours to be available for export. The export task can also take up to a few hours to process. This process depends on the size of the data that you export. It's a best practice to use log group-level subscription filters to process in real time or continuously archive new data to Amazon S3. For more information, see Real-time processing of log data with subscriptions.

Related information

Why is the log data that I exported from CloudWatch to Amazon S3 either missing or invalid?

How do I retrieve log data from CloudWatch Logs?

Exporting log data to Amazon S3