By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

EventBridge Pipe stops after 24 hour with error "PROBLEM: Pipe VPC event source require outbound internet access to send events to Pipes"

0

I have a public reachable Self-Managed Kafka server, sitting on a public subnet. I have created an EventBridge pipe that connects to a topic, reads messages and sends them to an SQS queue. After 24 hours the pipe is stopped with the error "PROBLEM: Pipe VPC event source require outbound internet access to send events to Pipes". If I start the Pipe again and send messages to the topic, it simply doesn't work anymore. I can see the consumer group created by the Pipe, and the lag value keeps on increasing, this means the Pipe died and doesn´t read anything anymore.

My only option is to delete the pipe and re-create it. I have tried creating the same pipe against a private IP ont he same kafka server. But it doesn't work. Even if the pipe is attached to all subnets. Of course I could try now to create an MSK or private Self-managed server and spend hours chasing the issue, assuming the network causes the Pipe to die. But shouldn't the Pipe simply recover itself?

Topics
ServerlessApplication IntegrationAnalytics
Tags
Amazon EventBridgeAmazon Managed Streaming for Apache Kafka (Amazon MSK)
Language
English
Boris Janischevsky
asked a month ago114 views
2 Answers
0

Hello.

I have tried creating the same pipe against a private IP ont he same kafka server.

If you are trying to configure it for a private IP, are you using a private subnet or something?
In that case, it seems that a NAT gateway is required, but have you already set it up?
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-msk.html#pipes-msk-vpc-config

profile picture
EXPERT
Riku_Kobayashi
answered a month ago
profile picture
EXPERT
Giovanni Lauria
reviewed a month ago
  • Boris Janischevsky
    a month ago

    I got a standard VPC, with Internet Gateway, nat gateway, 4 subnets, 2 public, 2 private, all works perfectly well, until the next morning. The Kafka EC2 server is on a public subnet with public IP address. I can access it remotely and locally from resources on the VPC. Lambdas can access the server through its private IP (although the server only responds to the public one). Eventbridge fails to conect to the private IP (it doesn´t do anything, it stalls, not even the consumer group gers created), so it is setup to use the public IP. Nevertheless, the Pipe shouldn´t suddenly stop after running for a day.

    I can sent you the VPC network diagram, but it´s all standard stuff created by the VPC wizard.

    I have setup a parallel SelfManaged Kafka server that´s only attached to a private subnet. Created now two pipes, one against the Public server, one against the private server. Both work fine. Will wait until tomorrow and publish a verdict here.

  • Boris Janischevsky
    a month ago

    Both pipes failed again in 24 hours. I had both pipes attached to all 4 subnets. I will attach now each pipe to only one subnet, the private Kafka pipe to the private subnet and the public Kafka pipe I will attach to the public subnet.

0

I´ll answer to myself here:

  • I tested now side-by-side a SelfManaged Kafka server on a private subnet and public subnet.
  • Every time I build a pipe (or a rule) in EventBridge that reads from Kafka, whichever of the Kafka servers, if I:
    • attach only the public subnet it fails to create the consumer group, although the pipe or rule says it's active. Full debugging of a pipe with cloudWatch traces, says nothing. I then have to delete the pipe or rule and start from scratch. If I attach private subnet it simply doesn't work.
    • if I attach only a private subnet being this for the public or private kafka sever everything works fine and the rule/pipe seems to never break.
    • if I attach a combination of public/private subnets, it´s a matter of time until something breaks.

Conclusion: **do only attach to private subnet your rule/pipe to a private subnet. **

Boris Janischevsky
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content