msk connector not able to fetchdata from msk cluster.. timeout error INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:23

Question

INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1746775156865, tries=1, nextAllowedTryMs=1746775156966) timed out at 1746775156866 after 1 attempt(s)

We are trying to setup questdb connector with msk..

We have IAM enabled for msk cluster... 
Following configuration is required when from local trying to create topics/ producer/consume messages.. 
```
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

now when we are trying to create the questdb connector , in Configuration settings , we are adding the above configs .. 
but still getting error : INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235) org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1746775156865, tries=1, nextAllowedTryMs=1746775156966) timed out at 1746775156866 after 1 attempt

msk & connector are in same vpc... 
```
connector.class=io.questdb.kafka.QuestDBSinkConnector
topics=test2
tasks.max=10
include.key=false
client.conf.string=http::addr=:9000;username=admin;password=123;
name=connector2questdb4
value.converter.schemas.enable=false
table=dtest_kf
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

Also following role is attached to the service execution role of the connector..

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka:DescribeCluster",
                "kafka:DescribeClusterV2",
                "kafka:GetBootstrapBrokers",
                "kafka:ListScramSecrets"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:WriteData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "*"
        }
    ]
}
```
for now we are giving resource *...

and trust policy

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "kafkaconnect.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "11111111"
                },
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:kafkaconnect:eu-west-2:11111111:connector/*"
                }
            }
        }
    ]
}

```

could someone give some idea.. how to resolve ?

in connector logs we are getting

```
2025-05-09T07:18:46.000Z
[Worker-0ec0caf65151e590e] [2025-05-09 07:18:46,847] INFO Kafka startTimeMs: 1746775126846 (org.apache.kafka.common.utils.AppInfoParser:121)
2025-05-09T07:19:16.000Z
[Worker-0ec0caf65151e590e] [2025-05-09 07:19:16,867] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
2025-05-09T07:19:16.000Z
[Worker-0ec0caf65151e590e] org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1746775156865, tries=1, nextAllowedTryMs=1746775156966) timed out at 1746775156866 after 1 attempt(s)
2025-05-09T07:19:16.000Z
[Worker-0ec0caf65151e590e] Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2025-05-09T07:19:46.000Z
[Worker-0ec0caf65151e590e] [2025-05-09 07:19:46,872] INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser:83)
2025-05-09T07:19:46.000Z
[Worker-0ec0caf65151e590e] [2025-05-09 07:19:46,873] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
2025-05-09T07:19:46.000Z
[Worker-0ec0caf65151e590e] org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1746775186866, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)`
```

EdbE · Accepted Answer

While the full answer above by AI was overkill, it did provide correct suggestion. In most cases, timeout exception is coming from missed incoming rules for security groups.

Since SGs were not addressed in the original question, the recommendation would be to check SG that attached to MSK, that it allows access to SG that is attached to a connector.

*Note*, if you try to create a connector from AWS console, it will use a group that is attached to MSK cluster of your choice, so you will need to ensure that the SG has incoming rule for **itself** on port 9098

re:Post Agent · Answer

The issue you're experiencing with your MSK connector not being able to fetch data from the MSK cluster appears to be related to connectivity or authentication problems. Here are some suggestions to help resolve the issue:

1. Network Configuration:
   - Ensure that the security groups for both the MSK cluster and the connector allow traffic on the necessary ports (typically 9092 for plaintext, 9094 for TLS, and 9098 for SASL).
   - Verify that the connector and MSK cluster are indeed in the same VPC and subnet, and that the route tables are correctly configured.

2. IAM Role and Permissions:
   - Your IAM role seems to have the necessary permissions, but you might want to specify the exact resource ARNs instead of using "*" for tighter security.
   - Make sure the IAM role is correctly attached to the connector.

3. Connector Configuration:
   - Double-check that the bootstrap server addresses in your connector configuration are correct and using the right protocol (e.g., b-1.your-cluster-name.abcde.c1.kafka.us-east-1.amazonaws.com:9098).
   - Ensure that the aws-msk-iam-auth JAR file is in the connector's classpath.

4. Authentication:
   - Verify that the IAM credentials being used by the connector are valid and have not expired.
   - Make sure the connector is using the correct AWS region.

5. Logging and Debugging:
   - Enable debug logging for the connector to get more detailed information about the connection attempts.
   - Use AWS CloudTrail to check for any authorization failures.

6. Connectivity Test:
   - Try running a simple Kafka consumer or producer from the same VPC as the connector to verify basic connectivity to the MSK cluster.

7. MSK Cluster Health:
   - Check the health of your MSK cluster in the AWS console to ensure it's running correctly.

If these steps don't resolve the issue, you may need to contact AWS support for further assistance, especially if you suspect there might be an issue with the MSK cluster itself.  
**Sources**  
[Connect to AWS MSK cluster from Local mac | AWS re:Post](https://repost.aws/questions/QUW0c-V5bSQYKB5iRNguIhiQ/connect-to-aws-msk-cluster-from-local-mac)  
[Build multi-Region resilient Apache Kafka applications with identical topic names using Amazon MSK and Amazon MSK Replicator | AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/build-multi-region-resilient-apache-kafka-applications-with-identical-topic-names-using-amazon-msk-and-amazon-msk-replicator/)  
[PUT ACL to MSK cluster | AWS re:Post](https://repost.aws/questions/QUu95a0UiASJWj9kNRTD1qBA/put-acl-to-msk-cluster)

msk connector not able to fetchdata from msk cluster.. timeout error INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:23

Relevant content