"Topic not present in metadata after 60000 ms" on Amazon MSK: it may be an authorization failure, not a missing topic

Symptoms

A producer writing to Amazon MSK fails with:

org.apache.kafka.common.errors.TimeoutException: Topic orders.events not present in metadata after 60000 ms.

Read literally, the message says the topic does not exist. But when you check:

• The topic exists on the cluster (confirmed with kafka-topics.sh --list)
• The cluster is healthy (ActiveControllerCount is 1, OfflinePartitionsCount is 0)
• Other applications on the same cluster produce and consume normally
• Nothing changed in the application code or configuration

With this combination, the likely cause is not a missing topic. It is an authorization failure.

Cause: without DESCRIBE permission, the topic becomes invisible

Before sending messages, a Kafka producer calls the Metadata API to look up the partitions and leaders of the target topic. If the client does not have DESCRIBE permission on that topic, the broker does not return the topic's metadata. This is by design: Kafka does not reveal the existence of a topic to a client that is not authorized to see it.

From the client's point of view, the topic never appears in metadata, so after waiting out max.block.ms (60 seconds by default) it throws exactly this error:

Topic ... not present in metadata after 60000 ms

An authorization failure disguises itself as a missing topic. That is why permissions are so often missing from the list of suspected causes for this error.

On MSK clusters with IAM access control, the permission required for the Metadata call is kafka-cluster:DescribeTopic. See the semantic action-to-API mapping in the MSK IAM access control documentation.

The typical scenario where this appears is a deployment that introduces new topics. The application starts using a new topic, but the IAM policies of the involved roles still list only the pre-existing topics in their Resource entries. Since neither the code nor the existing topic permissions changed, it looks like "nothing changed but it suddenly fails."

Confirm it from the broker logs: look for CLIENT_NOT_AUTHORIZED

Instead of guessing, you can confirm the cause from broker logs. If broker log delivery is enabled on the cluster (CloudWatch Logs, S3, or Firehose), IAM authorization denials are recorded there.

In CloudWatch Logs Insights, select the broker log group and run:

fields @timestamp, @logStream, @message
| filter @message like /CLIENT_NOT_AUTHORIZED/
| sort @timestamp desc
| limit 100

If permissions are the problem, you will see entries like this (values replaced with examples):

[2026-01-15 09:00:00,123] INFO [...]: [CLIENT_IAM] [IAM]:[arn:aws:sts::123456789012:assumed-role/my-app-role/aws-sdk-java-1700000000000]:[10.0.1.10:9098-10.0.2.20:51538-1234]:[...] denied to perform DESCRIBE on resource orders.events of type TOPIC during API call Metadata because CLIENT_NOT_AUTHORIZED (kafka.authorizer.logger)

This one line contains everything you need:

• The denied principal: assumed-role/my-app-role — which IAM role
• The client IP: 10.0.2.20 — which instance or pod
• The denied operation and resource: DESCRIBE on resource orders.events of type TOPIC
• The API involved: during API call Metadata — the producer's metadata lookup
• The reason: CLIENT_NOT_AUTHORIZED

If the timestamps of these entries line up with your application's errors, the diagnosis is confirmed.

Tip: To narrow the search to one topic, add its name to the filter: | filter @message like /orders.events/ and @message like /denied/

Fix: add the topic resources to the IAM policy

Add permissions for the topic resources to the IAM policy of the denied role. For a producer, the minimal shape is:

{
  "Effect": "Allow",
  "Action": [
    "kafka-cluster:Connect",
    "kafka-cluster:DescribeTopic",
    "kafka-cluster:WriteData"
  ],
  "Resource": [
    "arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/<cluster-uuid>",
    "arn:aws:kafka:us-east-1:123456789012:topic/my-cluster/<cluster-uuid>/orders.events*"
  ]
}

• For consumers, use kafka-cluster:ReadData instead of WriteData, plus group resources (kafka-cluster:AlterGroup, DescribeGroup)
• The trailing wildcard on the topic name (orders.events*) covers derived topics (such as orders.events.internal) in one entry. If your topic naming is prefix-based, wildcards reduce the maintenance burden
• Note that Connect applies to the cluster resource, while DescribeTopic and WriteData apply to topic resources

Policy changes take effect on the next Metadata request without restarting the client. To verify, run the Logs Insights query again and confirm that CLIENT_NOT_AUTHORIZED entries for that role no longer appear.

Other causes of the same error, and how to tell them apart

This error only means "metadata did not arrive in time," so causes other than permissions exist. The broker logs are the differentiator.

What you see in broker logs	Direction
CLIENT_NOT_AUTHORIZED with during API call Metadata	Authorization failure (this article)
Repeated Failed authentication	Authentication failure (credentials, SASL configuration)
No entries at all for the client's IP	Network/DNS/bootstrap configuration (client never reaches the broker)

For the third case, check security groups, the bootstrap server string, and whether the port matches the authentication method (IAM: 9098, SASL/SCRAM: 9096, TLS: 9094). For general connectivity issues, see the MSK troubleshooting documentation.

Prevention: a checklist for deployments that add topics

This problem almost always appears in deployments that introduce new topics. Consider adding these to your pre-deployment checklist:

1. List every application role that will read from or write to the new topics (producers and consumers)
2. Verify that each role's IAM policy Resource entries cover the new topic ARNs (including wildcard coverage)
3. After the deployment, search broker logs for CLIENT_NOT_AUTHORIZED to catch omissions early

References

• Amazon MSK IAM access control
• Amazon MSK broker logs
• Amazon MSK troubleshooting
• Apache Kafka producer configs (max.block.ms)