By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to containerize kafka-kinesis-connector?

I have an on-prem data pipeline with MQTT + Kafka, each containerized locally. Now, I want to enable the upstream connection to the Cloud/Internet with AWS Kinesis, but I need a Kafka/Kinesis connector. ``` version: '3' services: nodered: container_name: nodered image: nodered/node-red ports: - "1880:1880" volumes: - ./nodered:/data depends_on: - mosquitto environment: - TZ=America/Toronto - NODE_RED_ENABLE_PROJECTS=true restart: always mosquitto: image: eclipse-mosquitto container_name: mqtt restart: always ports: - "1883:1883" volumes: - "./mosquitto/config:/mosquitto/config" - "./mosquitto/data:/mosquitto/data" - "./mosquitto/log:/mosquitto/log" environment: - TZ=America/Toronto user: "${PUID}:${PGID}" portainer: ports: - "9000:9000" container_name: portainer restart: always volumes: - "/var/run/docker.sock:/var/run/docker.sock" - "./portainer/portainer_data:/data" image: portainer/portainer-ce zookeeper: image: zookeeper:3.4 container_name: zookeeper ports: - "2181:2181" volumes: - "zookeeper_data:/data" kafka: image: wurstmeister/kafka:1.0.0 container_name: kafka ports: - "9092:9092" - "9093:9093" volumes: - "kafka_data:/data" environment: - KAFKA_ZOOKEEPER_CONNECT=10.0.0.129:2181 - KAFKA_ADVERTISED_HOST_NAME=10.0.0.129 - JMX_PORT=9093 - KAFKA_ADVERTISED_PORT=9092 - KAFKA_LOG_RETENTION_HOURS=1 - KAFKA_MESSAGE_MAX_BYTES=10000000 - KAFKA_REPLICA_FETCH_MAX_BYTES=10000000 - KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000 - KAFKA_NUM_PARTITIONS=2 - KAFKA_DELETE_RETENTION_MS=1000 depends_on: - zookeeper restart: on-failure cmak: image: hlebalbau/kafka-manager:1.3.3.16 container_name: kafka-manager restart: always depends_on: - kafka - zookeeper ports: - "9080:9080" environment: - ZK_HOSTS=10.0.0.129 - APPLICATION_SECRET=letmein command: -Dconfig.file=/kafka-manager/conf/application.conf -Dapplication.home=/kafkamanager -Dhttp.port=9080 volumes: zookeeper_data: driver: local kafka_data: driver: local ``` I found this one from your labs: https://github.com/awslabs/kinesis-kafka-connector Again, I run everything from a docker-compose and that works, but now I'm not sure if there's either an existing image or documentation that can help me figure out how to containerize this connector. Will I have to create my own custom image via a DockerFile? Any examples? Thank you.
1
answers
0
votes
17
views
asked 5 days ago

AWS MSK connector for datagen source is failing without errors

Hi , We have a msk cluster provisioned and try to create custom connector datagen source for s3, but its failing without errors, we can only see below logs in broker logs: [2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-13 at offset 0 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-19 at offset 0 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-7 at offset 0 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. [2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-1 at offset 0 (kafka.server.ReplicaFetcherThread) org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
0
answers
0
votes
17
views
asked a month ago

MSK Connect Postgres connector fails during getting Glue avro schema details

I have problem using MSK Postgres Debezium connector with Glue Schema Registry avro serialisation, and getting "connect timed out" to GSR. Logs are following: ``` [Worker-051272e114b69c525] [2022-08-17 08:47:55,387] ERROR [route-events-connector|task-0] WorkerSourceTask{id=route-events-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:191) ... com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter.fromConnectData(AWSKafkaAvroConverter.java:97) [Worker-051272e114b69c525] at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63) [Worker-051272e114b69c525] at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:313) [Worker-051272e114b69c525] at com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter.fromConnectData(AWSKafkaAvroConverter.java:95) [Worker-051272e114b69c525] ... 15 more [Worker-051272e114b69c525] Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to get schemaVersionId by schema definition for schema name = key-schema com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:144) [Worker-051272e114b69c525] ... 28 more [Worker-051272e114b69c525] Caused by: java.net.SocketTimeoutException: connect timed out [Worker-051272e114b69c525] at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ``` Related to GSR connector config: ``` ... key.converter.region=eu-central-1 key.converter.registry.name=my-schema-registry key.converter.schemaAutoRegistrationEnabled=true key.converter.schemaName=key-schema key.converter.avroRecordType=GENERIC_RECORD key.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter value.converter.region=eu-central-1 value.converter.registry.name=my-schema-registry value.converter.schemaAutoRegistrationEnabled=true value.converter.schemaName=value-schema value.converter.avroRecordType=GENERIC_RECORD value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter ``` We have already configured MSK connect json Postgres connectors which are working fine and publishing data to MSK topics. Has anyone successfully configured MSK Connect with Glue Schema Registry for avro serialization? Thanks.
1
answers
0
votes
78
views
asked 2 months ago