How do I troubleshoot a failed export task in CloudWatch Logs?

5 minute read

I want to troubleshoot a failed export task in Amazon CloudWatch Logs.

Short description

You can configure an export task for an Amazon Simple Storage Service (Amazon S3) bucket within the same account or a different account. If the destination Amazon S3 bucket doesn't have the necessary permissions, then the following errors might appear:

Could not create export task. GetBucketAcl call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation.
Could not create export task. PutObject call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation.
Unable to create export task. Resource limit exceeded.
PutObject call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation. This may be a result of a KMS key or bucket encryption misconfiguration.
GetBucketAcl call on the given bucket failed. Please check if the specified Amazon S3 Bucket is in the same AWS region as CloudWatch Logs.

Resolution

To troubleshoot a failed export task in CloudWatch Logs, complete the following steps:

Grant permissions for the export task

To grant the necessary permissions for the export task, use the following permissions in your users, groups, or roles:

{
  "Version": "2012-10-17",
  "Statement": [
      {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-exported-logs/*"
    },
    {
      "Effect": "Allow",
      "Action": [
      "kms:GenerateDataKey",
      "kms:Decrypt"
       ],
    "Resource": "ARN_OF_KMS_KEY"
    }
  ]
}

To run the export task, make sure that you authenticate the AWS Identity and Acccess Management (IAM) role AmazonS3ReadOnlyAccess with the following permissions:

logs:CreateExportTask
logs:CancelExportTask
logs:DescribeExportTasks
logs:DescribeLogStreams
logs:DescribeLogGroups

Grant permissions on the destination Amazon S3 bucket

By default, all Amazon S3 buckets and objects are private. Only the resource owner or the AWS account that created the bucket can access the bucket and objects within the bucket. When necessary, the resource owner can write an access policy to grant access permissions to other resources and users.

Note: When you set a policy, it's a best practice to include a randomly generated string as the prefix for the bucket. This allows only the intended log streams to be exported to the bucket.

To grant permissions on the destination Amazon S3 bucket, complete the following steps:

Open the Amazon S3 console and choose the bucket that you want to export logs to.
Choose Permissions, and then Bucket policy.

In the Bucket Policy Editor, add the following policy:

Note: Replace example-s3-bucket with the name of your Amazon S3 bucket. Also replace example-region with the required AWS Region endpoint for the Principal parameter.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "s3:GetBucketAcl",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::exammple-s3-bucket",
      "Principal": {
        "Service": "logs.example-region.amazonaws.com"
      },
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": [
            "AccountId1",
            "AccountId2",
            ...
          ]
        },
        "ArnLike": {
          "aws:SourceArn": [
            "arn:aws:logs:Region:AccountId1:log-group:",
            "arn:aws:logs:Region:AccountId2:log-group:",
            ...
          ]
        }
      }
    },
    {
      "Action": "s3:PutObject",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::example-s3-bucket/",
      "Principal": {
        "Service": "logs.example-region.amazonaws.com"
      },
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control",
          "aws:SourceAccount": [
            "AccountId1",
            "AccountId2",
            ...
          ]
        },
        "ArnLike": {
          "aws:SourceArn": [
            "arn:aws:logs:Region:AccountId1:log-group:",
            "arn:aws:logs:Region:AccountId2:log-group:",
            ...
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::create_export_task_caller_account:role/role_name"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::example-s3-bucket/",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

Choose Save. The preceding policy is the set as the access policy on your Amazon S3 bucket. This policy allows CloudWatch Logs to export log data to your Amazon S3 bucket and the bucket owner has full permissions on all exported objects.

Export a task to a bucket encrypted with SSE-KMS

Open the AWS Key Management Service (AWS KMS) console. To change the AWS Region, use the Region selector in the upper-right corner of the page.
In the left navigation bar, choose Customer managed keys, choose Create Key, and then complete the following:
For Key type, choose Symmetric.
For Key usage, choose Encrypt and decrypt.
Choose Next.
Under Add labels, enter an alias for the key. Optionally, add a description or tags. Then, choose Next.
Under Key administrators, select who can administer this key, and then choose Next.
Under Define key usage permissions, make no changes, and then choose Next.
Review the settings, and then choose Finish.
Navigate to the Customer managed keys page, and then choose the name of the key that you just created.
Choose the Key policy tab, and then choose Switch to policy view.
In the Key policy section, choose Edit.

Add the following statement to the key policy statement list:

Note: Replace example-region with the Region of your logs. Replace example-account-arn with the account ARN that owns the AWS KMS key.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Allow CWL Service Principal usage",
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.example-region.amazonaws.com"
      },
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": ""
    },
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "example-account-arn"
      },
      "Action": [
        "kms:GetKeyPolicy",
        "kms:PutKeyPolicy",
        "kms:DescribeKey",
        "kms:CreateAlias",
        "kms:ScheduleKeyDeletion",
        "kms:Decrypt"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Enable IAM Role Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::create_export_task_caller_account:role/role_name"
      },
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "ARN_OF_KMS_KEY"
    }
  ]
}

Choose Save changes.
Open the Amazon S3 console.
Choose the bucket that you would like to export CloudWatch Logs to.
Choose the Properties tab. Then, under Default Encryption, choose Edit.
Under Server-side Encryption, choose Enable.
Under Encryption type, choose AWS Key Management Service key (SSE-KMS).
Choose Choose from your AWS KMS keys and find the key that you created.
For Bucket key, choose Enable.
Choose Save changes.

Related information

Export log data to Amazon S3 using the console

Topics

Management & Governance

Relevant content

create-export-task | Filter CloudWatch logs using JMESpath
username-lm-217
asked 2 years ago
How do I delete an export task from Cloudwatch to S3?
Tony
asked 12 days ago
Cloudwatch Export Task stuck on "Started successfully"
Caldera Staging
asked 6 months ago
Is it possible to get a single & sorted exported log data from CloudWatch Logs export task?
Accepted Answer
AWS-User-8775074
asked 2 years ago
Exporting Cloudwatch log group To S3 is failing
Accepted Answer
opsdev19004
asked 7 months ago
How do I troubleshoot CloudWatch logs that fail to export to S3 buckets?
AWS OFFICIALUpdated a month ago
How can I resolve issues exporting Amazon Aurora snapshots to Amazon S3?
AWS OFFICIALUpdated a year ago
I configured Amazon CloudWatch to export log data to Amazon S3, but the log data is either missing or invalid. How do I resolve this issue?
AWS OFFICIALUpdated 2 years ago
How do I export a Java heap dump file to an S3 bucket?
AWS OFFICIALUpdated 4 months ago
Configure CORS in Amazon S3
EXPERT
Gayathri Krishnamoorthy
published a year ago