How do I troubleshoot issues with my pipe in EventBridge Pipes?

Short description

To troubleshoot issues with your EventBridge pipe, first check your pipe's configuration. If you still encounter issues, then review your logs and EventBridge metrics to identify the issue and at what step it's occurring. Then, troubleshoot based on the issue that you identify.

Note: You can use EventBridge Pipes only for specific event sources.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Check your configuration for issues

Take the following actions:

• Verify that the source Amazon Resource Name (ARN) is correct.
• For the Amazon Kinesis source, make sure that the starting position is TRIM_HORIZON or AT_TIMESTAMP and within the retention period. For Amazon DynamoDB, make sure that it's TRIM_HORIZON and within the retention period.
  Note: If you specify LATEST as the starting position, then the pipe might miss events that are sent during pipe creation or updates.
• Make sure that you adhere to the maximum batch size for your source and target.
  Note: If you combine sources and targets, then the maximum batch size is the maximum size of the target. For more information, see Amazon EventBridge Pipes batching and concurrency.
• Make sure that you configured the correct target ARN.
• Make sure that the target service is available in your AWS Region.

Identify where the issue occurred

Configure logging

Important: Execution data might contain sensitive information from your event payloads. For more information, see the Including execution data in EventBridge Pipes logs.

To use the console to activate logging, complete the following steps:

1. Open the EventBridge console.
2. In the navigation pane, choose Pipes.
3. Select your pipe, and then choose Edit.
4. Choose the Pipe settings tab.
5. Under Logging, for Log level, choose a log level.
   Note: To get detailed execution information such as event payloads, enrichment responses, and filter results, activate the TRACE log level.
6. For Log destination, select one or more log destinations.
   Note: Your AWS Identity and Access Management (IAM) execution role must have the required permissions to write to Amazon CloudWatch Logs, Amazon Simple Service Storage (Amazon S3), or Amazon Data Firehose.
7. (Optional) To log event payloads, AWS requests, and responses, select Include execution data.
8. Choose Save changes.

Or, run the following update-pipe AWS CLI command to activate logs:

aws pipes update-pipe \
  --name pipe-name \
  --region region \
  --role-arn iam-role-arn \
  --log-configuration '{
    "Level": "INFO",
    "CloudwatchLogsLogDestination": {
      "LogGroupArn": "arn:aws:logs:region:account-id:log-group:log-group-name"
    },
    "S3LogDestination": {
      "BucketName": "bucket-name",
      "BucketOwner": "account-id",
      "OutputFormat": "json",
      "Prefix": "prefix-name/"
    },
    "FirehoseLogDestination": {
      "DeliveryStreamArn": "arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
    },
    "IncludeExecutionData": ["ALL"]
  }'

Note: Replace pipe-name with your EventBridge Pipe name, region with your Region, and iam-role-arn with the execution role's ARN. Also, replace account-id with your AWS account ID. The preceding example command activates logging for all three destinations. Replace log-group-name with your log group, bucket-name with your Amazon S3 bucket, prefix-name with your S3 bucket prefix, and delivery-stream-name with your delivery stream.

To identify the step where your Pipe execution failed in your log records, complete the following steps:

1. Open your logs.
2. Search for EXECUTION STARTED, EXECUTION_FAILED, ENRICHMENT_INVOCATION_FAILED, or TARGET_INVOCATION_FAILED entries:
   If EXECUTION_FAILED appears, then the target doesn't have the required permissions or there are configuration errors. Check for issues with your pipe's configuration.
   If EXECUTION_STARTED followed by TARGET_INVOCATION_FAILED appear, then there's an issue with the target invocation. Make sure that you configured the required IAM permissions for the pipe's execution role. If the target uses a resource-based policy, then also verify that the policy includes the required permissions.
   If ENRICHMENT_INVOCATION_FAILED appears, then the issue occurred in the enrichment step. Check the service logs for your enrichment type. For AWS Lambda, check CloudWatch Logs. For AWS Step Functions, check the execution history. For Amazon API Gateway, check CloudWatch Logs or the API destination logs.
   If EXECUTION_THROTTLED appears, then you exceeded your service quotas. Request a quota increase.
3. Check the error message that's associated with the failed step to determine the root cause.
4. Check the time that the error occurred to identify whether it's related to a specific event.

Note: The maximum size for log records is 256 KB. The log destination truncates fields that exceed this quota.

Check your pipe's metrics and state

To check your EventBridge pipe's metrics, complete the following steps:

1. Open the Amazon CloudWatch console.
2. In the navigation pane, choose Metrics, and then choose All metrics.
3. For Metrics, select the EventBridge/Pipes namespace.
4. To view all metrics, choose Across All Pipes. To view metrics for a specific pipe, choose By Pipe Name.
5. Check the ExecutionFailed, EnrichmentStageFailed, TargetStageFailed, and Duration metrics.

Then, complete the following steps to check the pipe state and identify why the state failed:

1. Open the EventBridge console.
2. In the navigation pane, choose Pipes.
3. Select your pipe name.
4. Check the values for State and State reason for detailed error information.

Or, run the following describe-pipe AWS CLI command to check the pipe's state:

aws pipes describe-pipe --name my-pipe-name

Note: Replace my-pipe-name with your pipe name.

Troubleshoot issues that you identified

Troubleshoot the "VPC event source requires outbound internet access" error

If the event source can't connect to the pipe, then you receive the "requires outbound internet access" error message.

To troubleshoot this issue, create Amazon Virtual Private Cloud (Amazon VPC) endpoints for the following services:

• For EventBridge Pipes, create an endpoint for com.amazonaws.region.pipes-data.
• For AWS Security Token Service (AWS STS), create an endpoint for com.amazonaws.region.sts.
• (SASL/SCRAM authentication only) For AWS Secrets Manager, create an endpoint for com.amazonaws.region.secretsmanager.
• For Lambda, create an endpoint for com.amazonaws.region.lambda.

If you don't use VPC endpoints, then create a NAT gateway in a public subnet to allow outbound internet access.

Verify that your pipe's security groups allow inbound traffic on the port for your event source and outbound traffic on port 443 to all destinations. If you use VPC endpoints, then also allow inbound traffic on port 443.

For more information, see Network configuration.

Troubleshoot "AccessDenied" errors and other access issues

If you receive "AccessDenied" errors, then make sure that your pipe's execution role has the required permissions for your source type.

Example policy for stream sources, such as Amazon Kinesis Data Streams or Amazon DynamoDB:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:ListStreams",
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:kinesis:region:account-id:stream/stream-name"
    }
  ]
}

Example policy for Amazon Simple Queue Service (Amazon SQS):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:region:account-id:queue-name"
    }
  ]
}

Example policy for sources that use customer managed keys that you encrypted with an AWS Key Management Service (AWS KMS) key:

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt"
  ],
  "Resource": "arn:aws:kms:region:account-id:key/key-id"
}

Troubleshoot authentication and authorization issues

If there are permissions issues between the pipe and its broker, then you receive an error message that's similar to the following example:

"PROBLEM: Cluster failed to authorize Pipes"

If you use SASL/SCRAM authentication, then take the following actions:

• Store your credentials in Secrets Manager.
• Make sure that the encryption role has the secretsmanager:GetSecretValue permission.
• If you encrypted the secret with AWS KMS, then make sure that the encryption role has kms:Decrypt permission.
• Choose the secret key in your pipe's source configuration.

If you use Amazon Managed Streaming for Apache Kafka (Amazon MSK), then make sure that the execution role has kafka-cluster:Connect, kafka-cluster:DescribeGroup, and kafka-cluster:ReadData permissions. Also, check the IAM policy to verify that the Resource uses the correct ARN for the cluster.

If you use Mutual TLS (mTLS), then take the following actions:

• Store the client certificate and private key in Secrets Manager.
• When you configure the pipe source, make sure that you use the correct certificate ARN.
• Verify that the certificate is valid and not expired.

For more information, see Introducing mutual TLS authentication for Amazon MSK as an event source.

Troubleshoot pipes that don't filter events as expected

If the pipe doesn't filter events how you expected, then take the following actions:

• Use event pattern syntax to filter your events.
• Test filter patterns with sample events before you deploy them.
  Important: Filters apply to the entire event payload structure. This filtering includes nested fields.
• In your logs, activate the TRACE log level to view the events that the pipe filtered.
• Check the filter and nested fields to make sure that the filter syntax matches your source event structure.
• In the JSON file, check for syntax errors such as missing commas or brackets, unsupported comparison operators, mismatched field names, or incorrect nesting.
  Important: Field patterns are case-sensitive.

Troubleshoot pipe executions that time out or experience high latency

Sources and targets have different batch size quotas that you must adhere to.

Make sure that the batch is less than 10,000 records, for a Kinesis Data Streams source or DynamoDB Streams source. For target quotas, see Supported batchable targets.

Also, make sure that enrichment responses are less than 6 MB and the batch window is less than 300 seconds. If needed, then adjust the batch window to balance latency and throughput. This setting controls how long EventBridge Pipes waits to accumulate records before it processes a batch.

Important: For Amazon SQS queues with low traffic, EventBridge might wait for up to 20 seconds before it invokes your pipe. This occurs even when you set a batch window to lower than 20 seconds. For more information, see Amazon SQS as a source in EventBridge Pipes.

To prevent timeouts, take the following actions:

• Make sure that enrichment functions complete within the maximum execution time of 15 minutes.
• If the target services can't process batches within the time quota, then reduce the batch size.
• If your system processes large messages, then reduce the batch size to decrease latency.
• Monitor the pipe's Duration, EventSize, EventCount, and ExecutionTimeout metrics in CloudWatch to identify performance bottlenecks.

For more information about EventBridge Pipes quotas, see Amazon EventBridge Pipes endpoints and quotas.

Troubleshoot events that fail during transformation

Check the pipe's EnrichmentStageFailed, TargetStageFailed, TargetStagePartiallyFailed, or ExecutionFailed metrics in CloudWatch to identify issues during the filtering or transformation stage. Issues occur during these stages when you exceed the payload or batch size quota.

To resolve this issue, reduce the Batch size for your EventBridge source, and create filters that occur before enrichment to reduce the payload size.

EventBridge applies input transformers to each JSON file in the array, not to the array as a whole. If enrichment returns an array that's larger than the target's batch size, then the pipe doesn't invoke the target. To identify JSON responses that aren't valid, check the EnrichmentTransformationFailed metric. In the JSON file, check for syntax errors such as missing commas or brackets, unsupported comparison operators, mismatched field names, or incorrect nesting. For more information, see Amazon EventBridge Pipes input transformation.

Important: Field patterns are case-sensitive.

For Lambda or Step Functions targets, verify that the returned array doesn't exceed the payload size quota.

Troubleshoot missing events or missing retries

Check whether there are missing events in the target destination, or there's a difference in the number of events in the target or source.

To verify that EventBridge didn't capture failed events or retry the events, check for data points in the ExecutionFailed, TargetStageFailed, or EnrichmentStageFailed metrics. Or, check your TRACE logging for repeat failures without successful retries.

To resolve this issue, modify your retry policy and configure a dead-letter queue (DLQ) for your Pipe to capture failed events.

Note: DLQ behavior for pipes varies by source type.

Related information

Amazon EventBridge Pipes

EventBridge Pipes error handling and troubleshooting

Using Amazon EventBridge with interface VPC endpoints