How do I troubleshoot errors when I'm trying to create a connector using Amazon MSK Connect?

8 minute read
2

I get an error when I try to create a connector using Amazon Managed Streaming for Apache Kafka (Amazon MSK) Connect.

Short description

When you create a connector using MSK Connect, you might receive one of these error messages:

  • There is an issue with the connector Code: UnknownError.UnknownMessage: The last operation failed. Retry the operation.
  • Invalid parameter connectorConfiguration: The following required field is missing or has invalid value: tasks.max
  • Invalid parameter serviceExecutionRoleArn: A service linked role ARN cannot be provided as service execution role ARN
  • org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches...
  • org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
  • org.apache.kafka.common.errors.SaslAuthenticationException: Too many connects
  • org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to s3.eu-central-1.amazonaws.com:443 [s3.eu-central-1.amazonaws.com/52.219.47.235 ] failed: connect timed out
  • org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to firehose.us-east-2.amazonaws.com:443 [firehose.us-east-2.amazonaws.com/52.95.23.168 ] failed: connect timed out
  • Connection to node - 1 (broker endpoint) failed authentication due to : Access Denied
  • ERROR Connection to node -3 (b-1.<cluster>.<region>.amazonaws.com/INTERNAL_IP) failed authentication due to: An error: (java.security .PrivilegedActionException: javax.security .sasl.SaslException: Failed to find AWS IAM Credentials [Caused by aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any ...........Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))

Resolution

There is an issue with the connector Code: UnknownError.UnknownMessage: The last operation failed. Retry the operation.

You get this error when MSK Connect can't create the connector, and the connector is moved to Failed state.

To find the root cause for the failure, review the log events for MSK Connect. MSK Connect writes log events that you can use to debug your connector. When you create a connector, you can specify zero log destinations or one of the following log destinations:

  • Amazon CloudWatch logs
  • Amazon Simple Storage Service (Amazon S3)
  • Amazon Kinesis Data Firehose

Invalid parameter connectorConfiguration: The following required field is missing or has invalid value: tasks.max

If you use a carriage return (/r) character at the end of a configuration, then you receive the following error:

Invalid parameter connectorConfiguration: The following required field is missing or has invalid value: tasks.max

To resolve this error, try the following troubleshooting steps:

  • Manually enter the configuration information in the connector configuration dialog box instead of copying and pasting from another source, such as documentation.
  • If you're on a Windows operating system, then use a text editor (such as Notepad++). You can use the text editor to check and remove the carriage return (CRLF) and any end-of-line (EOL) characters. To remove the carriage return, copy and paste the configuration into a text editor. From your text editor, choose View and Show Symbol. Then, choose Show All Characters to review any CRLF or EOL characters that might exist in a configuration. Replace all the CRLF characters ("\r\n") with LF characters ("\n").

Invalid parameter serviceExecutionRoleArn: A service linked role ARN cannot be provided as service execution role ARN.

You get this error when you use a service-linked role to create a connector. MSK Connect doesn't support using the service-linked role as the service execution role. You must create a separate service execution role. For instructions on how to create a custom AWS Identity and Access Management (IAM) role, see Creating a role to delegate permissions to an AWS service. Specify the role that you want the connector to work with. This role must be different from the service-linked role AWSServiceRoleForKafkaConnect that the service uses internally to create the connector resources.

org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches...

To resolve this error, try the following troubleshooting steps:

  • Remove any carriage return (/r) characters that might exist in the connector configuration.
  • If multiple files are required for the connector plugin, then include these files in your zipped file. The zipped file is used during the creation of the connector plugin. The JAR files in the zipped file must also have the expected file structure documented for that plugin. It's a best practice to turn on logs for MSK Connect and review the logs to confirm that the file structure is correctly set.

org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata

If the connector can't reach your MSK cluster, then you receive a TimeOutException error. To resolve this error, try the following troubleshooting steps:

org.apache.kafka.common.errors.SaslAuthenticationException: Too many connects

If your MSK cluster is running on a kafka.t3.small broker type with IAM access control, then be aware of the connection limit. The kafka.t3.small instance type accepts only one TCP connection per broker per second. When this connection limit is exceeded, your creation test fails. As a result, you receive a SaslAuthenticationException error, indicating an invalid credentials error. For more information about MSK clusters and IAM access control, see How Amazon MSK works with IAM.

To resolve the SaslAuthenticationException error, take one of the following actions:

  • In your MSK Connect worker configuration, update the values for reconnect.backoff.ms and reconnect.backoff.max.ms to "1000" or higher.
  • Upgrade to a larger broker instance type (such as kafka.m5.large or higher). For more information about Amazon MSK broker types and choosing the right broker type, see Broker types and Right-size your cluster.

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to s3.us-east-1.amazonaws.com:443 failed: connect timed out

You get this error when the connector can't connect to Amazon S3. To troubleshoot this error, make sure that you created the Amazon Virtual Private Cloud (Amazon VPC) endpoint from the cluster's VPC to Amazon S3. To create an Amazon VPC endpoint from the cluster's VPC to Amazon S3, complete the following steps:

  1. Open the Amazon VPC console.
  2. In the navigation pane, choose Endpoints.
  3. Choose Create endpoint.
  4. For Service category, choose AWS services.
  5. Under Services, choose the Service Name filter, and then select com.amazonaws.<region>.s3. Replace <region> with your AWS Region.
    Choose the Type filter, and then choose Gateway.
  6. For VPC, select the cluster's VPC.
  7. Under Route tables, select the route table that's associated with the cluster's subnets.
  8. Choose Create endpoint.

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to firehose.us-east-2.amazonaws.com:443 failed: connect timed out

You get this error when the connector can't connect to Amazon Kinesis Data Firehose. To troubleshoot this error, make sure that you created the Amazon VPC endpoint from the cluster's VPC to Kinesis Data Firehose.

To create an Amazon VPC endpoint from the cluster's VPC to Kinesis Data Firehose, follow the steps from the preceding section. Use the Service name filter com.amazonaws.<region>.kinesis-firehose.

Connection to node - 1 (b1.<cluster>.<region>.amazonaws.com) failed authentication due to : Access Denied

You get this error when the IAM user for MSK Connect doesn't have the required permissions to create a connector.

When you create a connector with MSK Connect, you must specify an IAM role to use with it. Your service execution role must have the following trust policy so that MSK Connect can assume this role:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {
            "Service": "kafkaconnect.amazonaws.com"
        },
        "Action": "sts:AssumeRole",
        "Condition": {
            "StringEquals": {
                "aws:SourceAccount": "Account-ID"
            },
            "ArnLike": {
                "aws:SourceArn": "MSK-Connector-ARN"
            }
        }
    }]
}

If the MSK cluster that you want to use with your connector uses IAM authentication, then you must add the following permissions policy to the connector's service execution role:

{
    "Version": "2012-10-17",
    "Statement": [{
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "cluster-arn"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": [
                "ARN of the topic that you want a sink connector to read from"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:WriteData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": [
                "ARN of the topic that you want a source connector to write to"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": [
                "arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/__amazon_msk_connect_*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:region:account-id:group/cluster-name/cluster-uuid/__amazon_msk_connect_*",
                "arn:aws:kafka:region:account-id:group/cluster-name/cluster-uuid/connect-*"
            ]
        }
    ]
}

For information on finding your cluster's UUID and construct topic ARNs, see Resources.

ERROR Connection to node -3 (b-1.<cluster>.<region>.amazonaws.com/INTERNAL_IP ) failed authentication due to: An error: (java.security .PrivilegedActionException: javax.security .sasl.SaslException: Failed to find AWS IAM Credentials [Caused by aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any ...........Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))

You get this error when the IAM role that you used to create the connector doesn't have the required permissions.

Review the access policies and trust relationship of the IAM role for the connector that you use to access Amazon MSK. For more information, see Service execution role.

Related information

How do I connect to my Amazon MSK cluster using the Kafka-Kinesis-Connector?

MSK Connect

Troubleshooting your Amazon MSK cluster

AWS OFFICIAL
AWS OFFICIALUpdated a year ago
1 Comment

Regarding error messages below,

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to s3.eu-central-1.amazonaws.com:443 [s3.eu-central-1.amazonaws.com/52.219.47.235 ] failed: connect timed out

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to firehose.us-east-2.amazonaws.com:443 failed: connect timed out

If the connection has timed out with above exceptions though you have created Amazon VPC endpoint from the cluster's VPC to specific service, make sure that the security group of your MSK-C has outbound rules configured properly 1. Having an outbound (egress) rule to allow all ports on 0.0.0.0/0, will resolve the issue 2. If you would like to narrow down the IP-Address range in outbound rules, you need to specify the IP-Address range of specific service. To find the IP address ranges that specific service (like Amazon S3) uses, run below command, for example,

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.region=="us-east-1") | select(.service=="S3") | .ip_prefix'

54.231.0.0/17
52.92.16.0/20
52.216.0.0/15
profile pictureAWS
replied 5 months ago