I'm trying to create a connector using Amazon Managed Streaming for Apache Kafka (Amazon MSK) Connect. However, I receive an error message. How do I resolve this?
When you create a connector using MSK Connect, you might receive one of these error messages:
- "Invalid parameter connectorConfiguration: The following required field is missing or has invalid value: tasks.max"
- "org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches..."
- "org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata"
- "org.apache.kafka.common.errors.SaslAuthenticationException: Too many connects"
"Invalid parameter connectorConfiguration: The following required field is missing or has invalid value: tasks.max"
If you use a carriage return (/r) character at the end of a configuration, you receive the following error:
Invalid parameter connectorConfiguration: The following required field is missing or has invalid value: tasks.max
To resolve this error, try the following:
- Manually enter the configuration information in the connector configuration dialog box instead of copying and pasting from another source such as documentation.
- If you're on a Windows operating system, then use a text editor (such as Notepad++). You can use the text editor to check and remove the carriage return (CRLF) and any end-of-line (EOL) characters. To remove the carriage return, copy and paste the configuration into a text editor. From your text editor, choose View and Show Symbol. Then, choose Show All Characters to review any CRLF or EOL characters that might exist in a configuration. Replace all the CRLF characters ("\r\n") with LF characters ("\n").
"org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches..."
To resolve this error, try the following:
- Remove any carriage return (/r) characters that might exist in the connector configuration.
- If multiple files are required for the connector plugin, then make sure that these files are included in your zipped file. (The zipped file is used during the creation of the connector plugin.) The JAR files in the zipped file must also have the expected file structure documented for that plugin. It's a best practice to confirm that the file structure has been set correctly by reviewing and enabling logs for MSK Connect.
"org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata"
If the connector is unable to reach your Amazon MSK cluster, then you receive a TimeOutException error. To resolve this error, check the security group for your Amazon MSK cluster. The security group must allow inbound traffic from the client's security group (associated with Amazon MSK Connect). If an Amazon MSK cluster is being used, then make sure to add a rule that allows inbound traffic from itself. For more information about setting up these rules, see Security group rules.
"org.apache.kafka.common.errors.SaslAuthenticationException: Too many connects"
If your MSK cluster is running on a kafka.t3.small broker type with AWS Identity Access Management (IAM) access control, then be aware of the connection limit. The kafka.t3.small instance type accepts only one TCP connection per broker per second. When this connection limit is exceeded, your creation test fails. As a result, you receive an SaslAuthenticationException error, indicating an invalid credentials error. For more information about Amazon MSK clusters and IAM access control, see How Amazon MSK works with IAM.
To resolve the SaslAuthenticationException error, consider the following approaches:
- In your Amazon MSK Connect worker configuration, update the values for reconnect.backoff.ms and reconnect.backoff.max.ms to "1000" or higher.
- Upgrade to a larger broker instance type (such as kafka.m5.large or higher). For more information about Amazon MSK broker types and choosing the right broker type, see Broker types and Right-size your cluster.
How do I connect to my Amazon MSK cluster using the Kafka-Kinesis-Connector?
Troubleshooting your Amazon MSK cluster