When using MSK, the connection encountered this error @timestamp":"2024-02-28T11:08:51.795Z","caller":"kq/pusher.go:45"'content":"unexpected EOF: broker appears to be expecting TLS","level":"error"}

The error message "unexpected EOF: broker appears to be expecting TLS" suggests that there's an issue with TLS (Transport Layer Security) encryption when connecting to the Kafka brokers. It indicates that the broker is expecting a secure TLS connection, but the client is not providing it.

To resolve this issue, you need to ensure that your Kafka client is configured to use TLS encryption when connecting to the Kafka brokers. Here are some steps you can take to troubleshoot and fix the problem:

Verify TLS Configuration: Double-check your Kafka client configuration to ensure that it's configured to use TLS encryption. Make sure that the appropriate TLS certificates and keys are configured and accessible to the client.

Check Kafka Broker Configuration: Ensure that the Kafka brokers are configured to require TLS encryption for client connections. Verify the broker's SSL configuration settings to confirm that TLS encryption is enabled and properly configured.

Debug Connection: Enable debug logging in your Kafka client to get more detailed information about the connection process. This can help identify any specific TLS-related errors or issues that might be occurring during the connection process.

Validate Broker Addresses: Confirm that the Kafka broker addresses specified in your configuration are correct and accessible from your client environment. Ensure that the broker addresses match the actual addresses of your Kafka brokers.

Test Connectivity: Use tools like Kafka console utilities or Kafka client libraries to manually test connectivity to the Kafka brokers using TLS encryption. This can help isolate and diagnose any connection issues.

Review Security Policies: Check if there are any security policies or network configurations (e.g., firewalls, security groups) that might be blocking or interfering with TLS connections to the Kafka brokers.