org.apache.kafka.common.errors.TimeoutException: Topic my-topic-processing not present in metadata after 60000 m

Question

I have a MSK cluster in same VPC in which my EKS Kubernates services are running . on both sides ( MSK and EKS) Security groups are updated to allow traffic.

I am using Spring boot Kafka Library to publish messages to Kafka Topic and retrieve the response metadata using following codes. I am getting error as "org.apache.kafka.common.errors.TimeoutException: Topic my-topic-processing not present in metadata after 60000 m" while publishing the messages.

Application.Yaml
  kafka:
    bootstrap-url:
    bootstrap-servers: b-1..c2.kafka..amazonaws.com:9092
    producer:
      acks: 1
      properties:
        linger.ms: 5
        retry.backoff.ms: 100
        buffer.memory: 134217728
        request.timeout.ms: 10000
        delivery.timeout.ms: 30000

public void send(final @NonNull String topic,
                     final @NonNull Map headers,
                     final @NonNull String data) {
        var producerRecord = new ProducerRecord(topic, data);

final long startTime = System.currentTimeMillis();
        final CompletableFuture> future = kafkaTemplate.send(producerRecord);

future.whenCompleteAsync((result, ex) -> {
            long elapsedTime = System.currentTimeMillis() - startTime;
            if (ex != null) {
                log.error("Failed to send message to topic {}: {}", result.getRecordMetadata().topic(), ex.getMessage());
            } else {
                log.info("Sent message to topic {} partition {} with offset {} in {} ms",
                        result.getRecordMetadata().topic(),
                        result.getRecordMetadata().partition(),
                        result.getRecordMetadata().offset(),
                        elapsedTime);
            }
        });
}

Am i Missing any configuration in MSK to fetch the topic metadata?

Taz · Answer

The timeout means the Kafka producer can’t get metadata for the topic, which usually happens when the brokers aren’t reachable or the topic doesn’t exist. In MSK, brokers advertise their private DNS names, so your EKS pods must resolve and connect to those endpoints within the same VPC. Verify that your EKS cluster nodes’ security group allows outbound TCP 9092 and that the MSK security group allows inbound TCP 9092 from the EKS group. Confirm the topic exists by running `aws kafka list-topics --cluster-arn ` or using a Kafka client with `--list`. If it’s missing, create it using `kafka-topics.
—Taz

re:Post Agent · Answer

The error "org.apache.kafka.common.errors.TimeoutException: Topic my-topic-processing not present in metadata after 60000 ms" indicates that your Spring Boot application is trying to publish messages to a topic that doesn't exist in your MSK cluster.

Here are some steps to resolve this issue:

1. First, make sure the topic "my-topic-processing" actually exists in your MSK cluster. You need to create the topic before publishing messages to it. You can create a topic using the Kafka CLI tools:

```
kafka-topics.sh --create --topic my-topic-processing --bootstrap-server b-1..c2.kafka..amazonaws.com:9092 --partitions 3 --replication-factor 2
```

2. Check your connectivity between your EKS cluster and MSK:
- Verify that the security groups are properly configured to allow traffic on port 9092 (or the appropriate port you're using).
- Ensure that your EKS pods can reach the MSK brokers. You can test this by running a simple network connectivity test from within a pod.

3. Review your Kafka client configuration:
- Your bootstrap-servers configuration looks correct, but make sure you're using the right protocol (9092 for plaintext, 9094 for TLS, 9098 for IAM authentication).
- If your MSK cluster uses IAM authentication, you'll need additional configuration properties like:
```
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```
- You'll also need to include the aws-msk-iam-auth JAR in your classpath.

4. Consider increasing the timeout values in your configuration if network latency is an issue:
```
request.timeout.ms: 30000
metadata.max.age.ms: 300000
```

5. Enable more detailed logging for the Kafka client to get better insights into the connection issues.

Once you've created the topic and ensured proper connectivity between your EKS cluster and MSK, the timeout error should be resolved.  
**Sources**  
[Help me troubleshoot this problem. | AWS re:Post](https://repost.aws/questions/QUMVQFHrf4R4a7A11Aq-0YPA/help-me-troubleshoot-this-problem)  
[msk connector not able to fetchdata from msk cluster.. timeout error INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:23 | AWS re:Post](https://repost.aws/questions/QUmzCdTXUgSMSjlPj9qklklQ/msk-connector-not-able-to-fetchdata-from-msk-cluster-timeout-error-info-adminclient-clientid-adminclient-1-metadata-update-failed-org-apache-kafka-clients-admin-internals-adminmetadatamanager-23)

User-7083747 · Answer

The error depicts that the producer cannot find the topic using the provided Kafka configuration. The topic may exist, but the producer cannot access it due to authentication, connectivity, or partition mismatch. Mismatch in the partitions : the partition of the topic that the client is trying to write to does not exits. Furthermore, this issue may arise due to a version mismatch between the Kafka client dependency and the Spring Boot version being used. 
[+] https://spring.io/projects/spring-kafka.

Consider modifying the below setting in the custom configurations -

set the below to default values:
```
request.timeout.ms: 30000 
delivery.timeout.ms: 120000
```
Additionally,
```
metadata.max.idle.ms: 60000
metadata.max.age.ms: 60000
```

For Latency issues consider increasing the `batch.size` which defines the maximum number of bytes that will be included in one batch and fine tune the `linger.ms` - to control the amount of time a producer waits for a batch to fill. Smaller batches are computationally expensive for Kafka as they translate to more threads and I/O operations at once. 
[+] https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-kafka-client.html#bestpractices-kafka-client-performance

Additionally, try increasing partition count for the topic and then monitor your cluster heath and the partition count per broker 
[+] https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html

If you are encountering the same issue after verifying the above details, please reach out to us via support case so that we will be able to investigate the issue further.

org.apache.kafka.common.errors.TimeoutException: Topic my-topic-processing not present in metadata after 60000 m

Add your answer

Relevant content