connector is not able to connect to aws MSK

Question

Hi team,
I am working on case that using confluentinc-kafka-connect-s3-10.2.2.zip plugin to create connector in MSK connect.
The connector will connect to AWS MSK and save data to S3. 
Both the connector and MSK within the same VPC, and same security group. The security group is configured with "If an Amazon MSK cluster is being used, then make sure to add a rule that allows inbound traffic from itself."
However, I am still getting below error:  [same error for other brokers as well]

`[Worker-062e488d0b4551b47] [2022-11-12 00:05:28,340] INFO [AdminClient clientId=adminclient-1] Failed authentication with b-1.*.kafka.ap-southeast-2.amazonaws.com/INTERNAL_IP ([8e115136-72d6-48fb-a18b-6d9b8de34bb1]: Access denied) (org.apache.kafka.common.network.Selector:616)
[Worker-062e488d0b4551b47] [2022-11-12 00:05:28,343] ERROR [AdminClient clientId=adminclient-1] Connection to node -3 (b-1.*.kafka.ap-southeast-2.amazonaws.com/INTERNAL_IP) failed authentication due to: [8e115136-72d6-48fb-a18b-6d9b8de34bb1]: Access denied (org.apache.kafka.clients.NetworkClient:771)
[Worker-062e488d0b4551b47] [2022-11-12 00:05:28,345] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager:232)
[Worker-062e488d0b4551b47] org.apache.kafka.common.errors.SaslAuthenticationException: [8e115136-72d6-48fb-a18b-6d9b8de34bb1]: Access denied
[Worker-062e488d0b4551b47] [2022-11-12 00:05:28,362] INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser:83)
[Worker-062e488d0b4551b47] [2022-11-12 00:05:28,365] INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
[Worker-062e488d0b4551b47] org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1668211558346, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
[Worker-062e488d0b4551b47] Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata`

By the way, the aws MSK is enabled with SCRAM and IAM authentication; 
Default MSK worker configuration is applied.

Shall I config username/password in somewhere of connector or worker? or why the error comes?
Thanks,
Linwei

Answer

As per reference[1], if the Amazon MSK cluster that you want to use with your connector is a cluster that uses IAM authentication, then you must add the following permissions policy to the connector's service execution role:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": [
                "cluster-arn"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": [
                "ARN of the topic that you want a sink connector to read from"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:WriteData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": [
                "ARN of the topic that you want a source connector to write to"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopic"
            ],
            "Resource": [
                "arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/__amazon_msk_connect_*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:region:account-id:group/cluster-name/cluster-uuid/__amazon_msk_connect_*",
                "arn:aws:kafka:region:account-id:group/cluster-name/cluster-uuid/connect-*"
            ]
        }
    ]
}
```

I kindly request you to check if you have setup the IAM policy as mentioned above. If issue still persisits, then in order to dive deep into issue, we require details that are non-public information. Please open a support case with AWS using reference[2].

[1] https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-service-execution-role.html 
[2] https://console.aws.amazon.com/support/home#/case/create

Answer

Thanks @AkashD, that's right, I missed that part. and all good now.
Cheers

connector is not able to connect to aws MSK

관련 콘텐츠