Connect to AWS MSK cluster from Local mac

The error you're encountering suggests there might be several issues preventing your local machine from connecting to the AWS MSK cluster. Let's address these potential problems:

1. Authentication: Ensure that your AWS credentials are correctly set up on your local machine. You can do this by configuring the AWS CLI or setting environment variables for your AWS access key ID and secret access key.

2. Network Configuration: Check if your MSK cluster is configured for public access. By default, MSK clusters are only accessible within the same VPC. If you're trying to connect from outside the VPC, you need to enable public access for the cluster.

3. Security Group Settings: Verify that the cluster's security group allows inbound traffic from your local machine's IP address on the appropriate port (typically 9198 for public access).

4. IAM Permissions: Ensure that your IAM user or role has the necessary permissions to interact with the MSK cluster.

5. Classpath Configuration: Make sure the aws-msk-iam-auth JAR file is in your classpath when running Kafka commands. You can set this by using the CLASSPATH environment variable:

   export CLASSPATH=/path/to/aws-msk-iam-auth-1.1.5-all.jar
   

6. Client Properties: Your client.properties file looks correct, but ensure there are no extra spaces or line breaks, especially at the end of each line.

7. Broker Addresses: Use the correct bootstrap broker string for public access. This should be different from the private bootstrap broker string.

8. Firewall Settings: Check if your local firewall or any network security appliances are blocking the connection.

To troubleshoot, you can try running a telnet command to the broker's public endpoint on port 9198 to test basic connectivity:

telnet b-1.your-cluster-name.kafka.us-east-1.amazonaws.com 9198

If this connects successfully, it indicates that network connectivity is established, and the issue might be related to authentication or client configuration.

If you're still having issues after checking these points, you may need to review your Kafka client version compatibility with the MSK cluster version and ensure all necessary dependencies are correctly installed and configured.
Sources
Unable to access AWS MSK cluster using public endpoint from local machine | AWS re:Post
Connect to an Amazon MSK Provisioned cluster - Amazon Managed Streaming for Apache Kafka
Troubleshoot your Amazon MSK cluster - Amazon Managed Streaming for Apache Kafka
Help me troubleshoot this problem. | AWS re:Post