DMS - replicate CDC data from Aurora PostgreSQL Serverless to Amazon Kinesis Data Streams

Question

I am using AWS Database Migration Service (DMS) to replicate Change Data Capture (CDC) data from Aurora PostgreSQL Serverless to Amazon Kinesis Data Streams. My Aurora PostgreSQL source endpoint works fine, but when I try to test my Kinesis target endpoint, it fails with the following error:

"Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect to database."

Troubleshooting Steps Taken So Far
* The IAM role dms-kinesis-role attached to DMS has the following permissions
    {
  "Effect": "Allow",
  "Action": [
    "kinesis:PutRecord",
    "kinesis:PutRecords",
    "kinesis:DescribeStream",
    "kinesis:GetShardIterator",
    "kinesis:GetRecords",
    "kinesis:ListShards"
  ],
  "Resource": "arn:aws:kinesis:ap-south-1:4***********:stream/***"
}

* The role is trusted by DMS
* Verified Kinesis Stream Exists & is Active on the same region.
*  Also, my public subnet has NAT gateway, and I enabled public access in the DMS as well.

Are there any additional IAM permissions or settings required for Kinesis?

re:Post Agent · Answer

Based on the information provided and the error message you're encountering, there are a few additional considerations and steps you can take to resolve the issue with your AWS DMS to Kinesis Data Streams replication:

1. VPC Endpoint or Public Access: For AWS DMS version 3.4.7 and higher, when connecting to a Kinesis endpoint, you need to either configure DMS to use VPC endpoints or make your replication instance public. Since you mentioned enabling public access for DMS, ensure that your replication instance is indeed set to be public.

2. Security Group Configuration: Verify that the security group associated with your DMS replication instance allows outbound traffic to the Kinesis service. This is crucial for the connection to be established.

3. IAM Role Permissions: While the IAM permissions you've listed look correct for Kinesis operations, ensure that the role also has permissions to assume the role itself. You might need to add the "sts:AssumeRole" permission to the policy.

4. Kinesis Stream ARN: Double-check that the ARN specified in the Resource field of your IAM policy exactly matches your Kinesis stream's ARN, including the correct region and account ID.

5. Endpoint Configuration: When creating the Kinesis target endpoint in DMS, make sure you've correctly specified the ARN of the IAM role you created (dms-kinesis-role) in the endpoint settings.

6. Network Configuration: Ensure that the subnets associated with your DMS replication instance have proper routing to reach the Kinesis service, either through an internet gateway (for public subnets) or a NAT gateway (for private subnets).

7. Kinesis Service Access: If you're using a VPC endpoint for Kinesis, verify that it's properly set up in the same VPC as your DMS replication instance, with the correct subnets and security groups.

If these steps don't resolve the issue, you may want to check the DMS task logs for more detailed error messages. Additionally, consider using AWS CloudTrail to monitor API calls and identify any permission-related issues that might not be immediately apparent.  
**Sources**  
[Using Amazon Kinesis Data Streams as a target for AWS Database Migration Service - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html)  
[Tune replication performance with AWS DMS for an Amazon Kinesis Data Streams target endpoint – Part 3 | AWS Database Blog](https://aws.amazon.com/blogs/database/tune-replication-performance-with-aws-dms-for-an-amazon-kinesis-data-streams-target-endpoint-part-3/)

DMS - replicate CDC data from Aurora PostgreSQL Serverless to Amazon Kinesis Data Streams

Add your answer

Relevant content