dynamodb streams across accounts (and region at times) using lambda

Question

A customer is using dynamoDB streams to pass events to a Kafka topic.
They would like to use lambda to poll the stream, but the consumer (kafka cluster self managed in Kubernetis) is in a different account.
I see that the lambda that reads from DDB needs to be in the DDB account.

As a workaround customer is considering this-
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html
(deployed in the Kafka account in the kubernetis cluster) but would prefer have a serverless solution instead.

I found this-
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-cross-account-kinesis-stream/

Will this mean the traffic goes through public internet (not desired)
Is there a work around using VPC peering / VPC sharing / Private link

Note - The Kafka cluster can be in the same region, but sometime is in a different region so solution should support both.

Accepted Answer

I don't think that this is the right approach. The blogs you mention talk about triggering a Lambda function in one account from Kinesis/DDB streams in a different account.

I think that in your case the best would be to have a Lambda function in the first account (DDB) which is triggered from the stream. Let this function run in a VPC in the first account. Peer that VPC to the VPC that is hosting Kafka in the second account. The Lambda function should be able to talk via the peered connection and ingest records into Kafka.

If the two accounts are in the same Organization maybe by using VPC sharing there is no need to do VPC peering. I did not test it so I am not sure it works as Lambda can run in VPCs in the same account. Not sure how Shared VPC are treated.

Just to answer the question about having two lambdas, Lambda function runs by default in an AWS managed VPC. If that Lambda tries to invoke another Lambda, i think it will remain in the same VPC. The second Lambda will be running in the seconds account's VPC, so there will be no internet traffic as far as I can tell.

dynamodb streams across accounts (and region at times) using lambda

Relevant content