How do I troubleshoot the data feed for EC2 Spot Instances configured with Amazon S3?

Short Description

To check your charges for Amazon EC2 Spot Instances, configure a data feed that reports on your Spot Instances' usage and pricing. When you subscribe to the data feed, Amazon EC2 stores this data in an S3 bucket. If you have issues with your data feed, then complete the following troubleshooting steps, depending on your issue.

Resolution

Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

AWS KMS encryption

Your S3 bucket might be encrypted with a customer managed AWS Key Management Service (AWS KMS) key. In this case, you must allow service to delivery.logs.amazonaws.com within the AWS KMS key policy's permissions. This allows AWS KMS to deliver the data feed to the encrypted S3 bucket.

Include the following permissions in your key policy:

{
  "Sid": "Allow Logs Delivery to use the key",
  "Effect": "Allow",
  "Principal": {
    "Service": [
      "delivery.logs.amazonaws.com"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

For more information, see Amazon S3 bucket server-side encryption.

ACL permissions

When you subscribe to the data feed, your account receives access control list (ACL) permissions for your S3 bucket to receive the data feed. Without these permissions, the data feed stops delivery to the bucket. Make sure that these permissions still exist in your account:

1. Open the Amazon S3 console
2. Choose Buckets from the navigation pane, and then select the S3 bucket that's associated with your data feed.
3. Choose the Permissions tab.
4. Under Access control list (ACL), find the external account that's added as Grantee and subscribed to the data feed. Verify that this account has FULL_CONTROL permission for your S3 bucket. The ACL permission for AWS account awslogsdelivery is added as the canonical ID for the account:

   c4c1ede66af53448b93c283ce9448c4ba468c9432aa01d700d3878632f77d2d0

If the account doesn't have FULL_CONTROL, then it must subscribe to the data feed again to receive the necessary ACL permissions. To do this, run the create-spot-datafeed-subscription AWS CLI command.

Note: If you remove your data feed subscription, this doesn't automatically delete the associated ACL permissions. If you want to delete your data feed, then you must access the permissions in the Amazon S3 console (Steps 1-4) to manually delete them.

Data feed limit

You can create only one data feed per AWS account. To transfer the data feed to another S3 bucket, run the create-spot-datafeed-subscription AWS CLI command again with a different S3 bucket.

EC2 Spot Instance runtime

You receive a data feed only for the time that your EC2 Spot Instances run. For example, if you stop your Spot Instances for three hours, then your data feed also stops delivering data for those three hours. Your data feed automatically resumes when the Spot Instances launch again.

S3 object ownership

The object ownership settings for your data feed's S3 bucket must be Bucket owner preferred or Object writer. These settings allow ACL permissions for your S3 bucket. If you don't have either of these ownership settings on your S3 bucket, then you can't configure the data feed on your S3 bucket. If this happens, then you receive an error:

"An error occurred (InaccessibleStorageLocation) when calling the CreateSpotDatafeedSubscription operation: The specified bucket does not exist or does not have enough permissions."

For more information, see Controlling ownership of objects and disabling ACLs for your bucket.

Additional considerations

When you configure your data feed, keep the following points in mind:

• There's no additional cost for activating a data feed. However, you incur storage costs for the data feed storage in your S3 bucket.
• Data feed files usually arrive in your bucket once per hour. Typically, a single data file covers each hour of usage. For more information on file size and arrival time, see Spot Instance data feed.