How can I troubleshoot Lambda Kinesis Firehose integration issues?

5 minute read
0

I've integrated AWS Lambda with Amazon Kinesis Data Firehose to transform incoming source data and deliver the transformed data to destinations. However, the Lambda function didn't invoke or failed.

Short description

Lambda functions configured with Amazon Kinesis Data Firehose might fail due to:

  • Insufficient AWS Identity and Access Management (IAM) permissions
  • Lambda duration exceeding the maximum allowed timeout
  • Lambda throttling
  • Data transformation errors
  • Lambda function errors

Note: If your Lambda function invocation has a network timeout or fails because you reached the invocation limit, Kinesis Data Firehose retries the invocation three times. For more information, see Data transformation failure handling.

Resolution

If you haven't already done so, turn on Amazon CloudWatch logging for your Kinesis Data Firehose. You can view the specific error logs when the Lambda invocation for data transformation or data delivery fails. You can turn on Kinesis Data Firehose error logging when you create your delivery stream. The format of the log group name is /aws/kinesisfirehose/delivery-stream-name. The delivery-stream-name is the name of the delivery stream.

To identify why the Lambda function failed, check the Amazon CloudWatch Logs group for /aws/lambda/lambda-function-name. Failed data transformation records are delivered to the Amazon Simple Storage Service (Amazon S3) bucket as a backup in the processing-failed folder. The records in your S3 bucket also contain the error message for failed invocation. For more information about resolving Lambda invocation failures, see Data transformation failure handling.

Insufficient IAM permissions

"AssumeRoleAccessDenied - Access was denied. Ensure that the trust policy for the provided IAM role allows Kinesis Data Firehose to assume the role."

This error occurs when the Amazon Kinesis Data Firehose IAM role isn't allowed to assume “firehose.amazonaws.com". Make sure that the IAM trust policy has permissions to assume “firehose.amazonaws.com" similar to the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
     "Action": "sts:AssumeRole"
    }
  ]
}

"InvokeAccessDenied - Access was denied. Ensure that the access policy allows access to the Lambda function."

This error occurs when the Amazon Kinesis Data Firehose IAM role isn't allowed permission to invoke the Lambda function used for data transformation. Make sure that the IAM policy allows Amazon Kinesis Data Firehose to invoke the Lambda function similar to the following:

Note: Replace the resource ARN in this example with your variables.

{
  "Effect": "Allow",
  "Action": [
    "lambda:InvokeFunction",
    "lambda:GetFunctionConfiguration"
  ],
  "Resource": [
    "arn:aws:lambda:region:account-id:function:function-name:function-version"
  ]
}

Lambda duration exceeding the maximum allowed timeout

"Firehose encountered timeout errors when calling AWS Lambda. The maximum supported function timeout is 5 minutes."

This error occurs when the Lambda function takes more than 5 minutes to complete. To troubleshoot Lambda function timeout errors with Kinesis Data Firehose, see Data transformation failure handling.

Lambda throttling

"InvokeLimitExceeded - The Lambda concurrent execution limit is exceeded. Increase the concurrent execution limit."

This error occurs when the Lambda concurrent execution limit is exceeded. Review the Lambda concurrency metrics in the CloudWatch logs to determine your usage. To request an increase for concurrent executions, see Requesting a quota increase.

For more information, see Lambda function scaling and How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?

Data transformation errors

"JsonProcessingException - There was an error parsing returned records from the Lambda function. Ensure that the returned records follow the status model required by Kinesis Data Firehose."

This error occurs when the transformed records returned by Lambda aren't in the required format. All transformed records returned from Lambda must contain the parameters recordId, result, and data. Check the Lambda function code and CloudWatch logs to verify that these records contain these parameters.

Note: To intentionally drop a record, set the value to Dropped.

For more information, see Data transformation and status model.

"DuplicatedRecordId - Multiple records were returned with the same record ID. Ensure that the Lambda function returns unique record IDs for each record."

This error occurs if multiple transformed records returned by the Lambda function to Kinesis Firehose have the same record ID. The record ID is passed from Kinesis Data Firehose to Lambda during the function invocation. Check the Lambda function code and CloudWatch logs to verify that each record ID is unique.

Lambda function errors

"The Lambda function was successfully invoked but it returned an error result."

This error occurs when Amazon Kinesis Data Firehose was able to successfully invoke the Lambda function but the Lambda function execution failed due to an error. Check the data transformation CloudWatch metrics to confirm that Kinesis Data Firehose has invoked your function.

If Kinesis Data Firehose hasn't invoked your Lambda function, then check the invocation time to see if it's beyond the timeout parameter. Your Lambda function might require a greater timeout value or need more memory to complete. For more information about invocation metrics, see Invocation metrics.

"Exceeded maximum allowed payload size” or “LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413."

Amazon Kinesis Data Firehose invokes a Lambda function synchronously and the invocation payload limit is 6 MB for each request and response. The Lambda payload limit is a hard limit and can't be increased. Make sure that the buffer size doesn't exceed 6 MB for each request and response.

For more information, see Lambda quotas.

Related information

Why am I experiencing a data delivery failure with Kinesis Data Firehose?

AWS OFFICIAL
AWS OFFICIALUpdated a year ago