AWS MSK topic creation issue

Based on the error message and your IAM policy, it appears you're facing an authentication issue when trying to create a topic in your MSK Serverless cluster. The error "[5d1704e6-0d11-4005-841a-9bda7a6dfc5c]: Access denied" indicates that your IAM permissions might not be correctly configured.

Here are some potential issues and solutions:

1. Your IAM policy resource ARN for the cluster might not be correctly formatted. For MSK Serverless, ensure you're using the correct ARN format. The resource ARN in your policy appears to be for a specific broker node rather than the cluster itself.

2. Your policy is missing some necessary permissions. For creating topics, you should include:

• kafka-cluster:Connect
• kafka-cluster:CreateTopic
• kafka-cluster:DescribeCluster
• kafka-cluster:DescribeTopic
• kafka-cluster:AlterCluster (this is often required for ACL operations)

3. The resource ARN for topics should include the cluster name and potentially a wildcard for the topic name. Make sure your ARN pattern matches the actual cluster name.

Try updating your IAM policy to include these permissions and ensure the resource ARNs are correctly formatted. The cluster ARN should typically look like: "arn:aws:kafka:region:account-id:cluster/cluster-name/*"

Also, verify that your client configuration (client.properties) is correctly set up for IAM authentication with the appropriate SASL mechanism and handler classes.

If you continue to face issues after updating the IAM policy, you might want to use AWS CloudTrail to identify which specific API calls are failing, which can help pinpoint any remaining permission issues.
Sources
PUT ACL to MSK cluster | AWS re:Post
MSK Serverless - Not authorized to access group <groupName> error | AWS re:Post

Hello,

The latest provided policy has some issues please try the below solution I hope it will help to resolve your issue,

To create a topic both the arn should be modified as

arn:aws:kafka:<region>:<account-id>:cluster/<cluster-name>/*
arn:aws:kafka:<region>:<account-id>:topic/<cluster-name>/*

please follow the below full policy to create a topic:

   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "kafka-cluster:Connect",
            "kafka-cluster:DescribeCluster",
            "kafka-cluster:AlterCluster",
            "kafka-cluster:Describe*"
         ],
         "Resource": [
            "arn:aws:kafka:<region>:<account-id>:cluster/<cluster-name>/*"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "kafka-cluster:CreateTopic",
            "kafka-cluster:WriteData",
            "kafka-cluster:DescribeTopic"
         ],
         "Resource": [
            "arn:aws:kafka:<region>:<account-id>:topic/<cluster-name>/*"
         ]
      }
   ]
} 

Please verify the following once again:

1. The trust relationship of your IAM role should be:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

2. The aws-msk-iam-auth-{LATEST VERSION}-all.jar should be in kafka_$KAFKA_VERSION.tgz/libs directory
3. The client configuration (client.properties) should correctly set up for IAM authentication with the appropriate SASL mechanism and handler classes as per the below MSK Serverless Developer Guide

For detailed set up, refer to AWS MSK Serverless Developer Guide https://docs.aws.amazon.com/msk/latest/developerguide/serverless-getting-started.html

Thank You!

Here is an updated policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster",
        "kafka-cluster:AlterCluster",
        "kafka-cluster:Describe*"
      ],
      "Resource": [
        "arn:aws:kafka:us-east-1:637423394741:cluster/demo-cluster-1/abcdef12-3456-7890-abcd-ef1234567890"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:CreateTopic",
        "kafka-cluster:WriteData",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": [
        "arn:aws:kafka:us-east-1:637423394741:topic/demo-cluster-1/abcdef12-3456-7890-abcd-ef1234567890/*"
      ]
    }
  ]
}