How do I troubleshoot a data delivery failure between Amazon Data Firehose and Amazon S3?

Resolution

Check the DeliveryToS3.Success metric to make sure that Firehose has tried to put data to your S3 bucket. If the DeliveryToS3.Success metric value is consistently at zero, then take the following actions.

Review resource availability

Confirm the availability of the S3 bucket that's specified in your Firehose delivery stream. If you're using the data transformation feature, then confirm that the specified AWS Lambda function exists.

Review inbound data records

Check the Firehose IncomingRecords and IncomingBytes metrics to verify that data is successfully sent to your Firehose stream. If the IncomingRecords or IncomingBytes values are zero, then no records reach Firehose.

If the delivery stream uses a Kinesis data stream as its source, then check the IncomingBytes and IncomingRecords metrics for the stream source. Also, verify whether the delivery stream emits DataReadFromKinesisStream.Bytes and DataReadFromKinesisStream.Records metrics. For more information about these metrics, see CloudWatch metrics for data delivery.

If no data reaches Firehose, then the issue might be upstream. For a direct PUT operation, confirm that the PutRecord and PutRecordBatch APIs used to put records to Firehose are called correctly.

Review Firehose logs

Check that you turned on error logging for Firehose. If you turned off logging, then check the error logs for delivery failure. The error logs provide specific reasons for delivery failure and help you identify problem areas. The format of the log group is /aws/kinesisfirehose/delivery-stream-name.

Then, use the following permissions for your role:

"Action": [
    "logs:PutLogEvents"
],
"Resource": [
    "arn:aws:logs:region:account-id:log-group:log-group-name:log-stream:log-stream-name"
]

Note: Replace region with the AWS Region where your CloudWatch logs are located, account-id with your AWS account ID, log-group-name with your CloudWatch log group name, and log-stream-name with your log stream name. 

Review the IAM role permissions

Make sure that the AWS Identity and Access Management (IAM) role that's specified in your Firehose delivery stream has the correct permissions. Depending on the parameters you turned on for the data stream, the role requires a number of permissions. For more information, see Grant Firehose access to an Amazon S3 destination.

For Amazon S3 access, update your IAM policy with the following policy:

"Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
],     
"Resource": [       
    "arn:aws:s3:::bucket-name",
    "arn:aws:s3:::bucket-name/*"                         
]

Note: Replace bucket-name with your S3 bucket name. 

To allow data transformation of your Lambda function, update your policy with the following policy:

"Action": [
               "lambda:InvokeFunction",
               "lambda:GetFunctionConfiguration"
],
"Resource": [
    "arn:aws:lambda:region:account-id:function:function-name:function-version"
]

Note: Replace region with the AWS Region where your Lambda function is deployed, account-id with your AWS account ID, function-name with the name of your Lambda function, and function-version with the version number. 

For a Kinesis data stream that is listed as a source, update your policy similar to the following example:

"Action": [
                "kinesis:DescribeStream",
                "kinesis:GetShardIterator",
                "kinesis:GetRecords",
                "kinesis:ListShards"
],
"Resource": "arn:aws:kinesis:region:account-id:stream/stream-name"

Note: Replace region with the Region where your Kinesis stream is deployed, account-id with your account ID, and stream-name with the name of your Kinesis data stream.

Review the Firehose server-side encryption

Firehose supports Amazon S3 server-side encryption with AWS Key Management Service (AWS KMS) to encrypt data that's delivered to Amazon S3. To allow server-side encryption, update your IAM role policy with a policy similar to the following example:

"Action": [
               "kms:Decrypt",
               "kms:GenerateDataKey"
           ],
           "Resource": [
               "arn:aws:kms:region:account-id:key/key-id"
           ],
           "Condition": {
               "StringEquals": {
                   "kms:ViaService": "s3.region.amazonaws.com"
               },
               "StringLike": {
                   "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::bucket-name/prefix*"
               }
           }

Note: Replace region with the Region where KMS key is located, account-id with your account ID, and key-id with your KMS key ID.

Review the AWS KMS encrypted S3 bucket

Confirm that the IAM role for the Firehose delivery stream has the correct permissions. To deliver data to an Amazon S3 bucket that's encrypted with AWS KMS, the key policy must allow the Firehose IAM role. For more information, see How do I resolve the "Access Denied" error in Firehose when writing to an Amazon S3 bucket?

Review Lambda invocation

Confirm that the Lambda function specified in your delivery stream is available. If you deleted the Lambda function or it’s missing, then create a new Lambda function to invoke.

Check the Firehose ExecuteProcessingSuccess and Errors metrics to confirm that Firehose tried to invoke your Lambda function. If the invocation fails then check the Amazon CloudWatch log group at the /aws/lambda/functionname location to identify why Lambda doesn’t invoke. If there's a Lambda transformation and Lambda invokes the function, then check the invocation duration. If the duration exceeds the timeout parameter, then your invocation fails. For more information about invocation metrics, see Using CloudWatch metrics with Lambda.

If data transformation fails, then the unsuccessfully processed records are delivered to your S3 bucket in the processing-failed folder. The format of the records in Amazon S3 also contains the error message. For more information about data transformation failures, see Transform source data in Amazon Data Firehose.

Note: Your S3 bucket policy can also explicitly deny access through aws:SourceIp or aws:SourceVpce. To verify whether your S3 bucket policy explicitly denies access, look for the S3.AccessDenied error code in CloudWatch Logs.