By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Access Secrets using VPC EndPoint in Kafka Event-source in EventBridge Pipes

1

**Scenario: **

  • MSK Cluster is in private subnet of VPC-1 and I am trying to create an EventBridge Pipe in VPC2 that gets data from the MSK cluster
  • VPC-1 and VPC-2 are connected using VPC peering
  • Kafka authentication details are stored in Secrets Manager
  • I am using "Self managed Apache Kafka" for Event source as both VPCs are in differet AWS accounts.

It works if I configure NAT gateway in VPC2. Is it possible to achieve the same functionality using VPC Endpoints?

Problem

  • I have deleted NAT gateway and I have created VPC endpoint for Secrets Manager as Event Source will need to access Kafka authentication information.
  • Also, added STS VPC endpoint
  • This setup fails to start the EventBridge Pipe with error message "PROBLEM: Pipe VPC event source require outbound internet access to send events to Pipes"

EventSource Setup

Topics
Networking & Content DeliveryServerlessApplication IntegrationAnalyticsSecurity, Identity, & Compliance
Tags
Amazon VPCAmazon EventBridgeAmazon Managed Streaming for Apache Kafka (Amazon MSK)AWS Secrets Manager
Language
English
Anupam Gupta
asked 8 months ago574 views
1 Answer
1
Accepted Answer

Adding the answer for community:

EvenBridge supports VPC interface endpoints but not for MSK/Self-Managed Kafka /Amazon MQ. So this is not a network issue but rather service support.

We need route out to internet to be able to use "Self Managed Kafka Event Source" in EventBridge Pipe.

Anupam Gupta
answered 8 months ago
profile picture
EXPERT
Giovanni Lauria
reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content