[Solved] Connectivity Issues with Kinesis Firehose Preview Capability and RDS (CDC)

Question

**Resolved**: I discovered that the ELB created to configure the VPC Services cannot have a security group associated with it. I didn’t test with a security group completely open to 0.0.0.0/0 (both inbound and outbound), but previously, I had tried adding inbound rules to the security group for the security group used by the RDS, plus an inbound rule for port 443, as indicated in the AWS documentation. After recreating the ELB without an associated security group, it worked.

---

I’m testing the new Kinesis Firehose feature as described in the AWS documentation: [link](https://docs.aws.amazon.com/firehose/latest/dev/database-as-source-iceberg.html). However, it seems the VPC Endpoint automatically created by Kinesis Firehose is unable to connect to the RDS.

Here’s my setup:

I have an Aurora RDS cluster with an RDS Proxy configured. I created a user with all the necessary permissions as specified in the AWS documentation: [link](https://docs.aws.amazon.com/firehose/latest/dev/das-sql-procedure-create-psql.html).

To test the functionality, I set up an ELB with a Target Group pointing to the RDS Proxy IPs. This worked as expected, as the ELB successfully connected to the RDS Proxy. Next, I created a VPC Endpoint Service based on the ELB, configuring it as per the documentation, allowing the principal firehose.amazonaws.com. Everything was still working fine. To validate connectivity, I created a VPC Endpoint pointing to the VPC Endpoint Service and confirmed that I could connect to the RDS Proxy within the VPC’s private network.

The problem arose when I created a Delivery Stream in Kinesis Firehose, configuring it to use the VPC Endpoint Service I had set up. Kinesis Firehose automatically creates a VPC Endpoint and associates it with my VPC Endpoint Service, but the connection to the RDS Proxy fails. I can see in the logs that Kinesis Firehose tries to connect to the RDS Proxy, but it doesn’t succeed. The Firehose destination logs show the following error:

```
{
    "deliveryStreamARN": "arn:aws:firehose:sa-east-1::deliverystream/POSTGRESQL-ICE-0BZSR",
    "destination": "arn:aws:glue:sa-east-1::catalog",
    "deliveryStreamVersionId": 1,
    "message": "Unable to establish connection with the database, please check if endpoint and port are valid.",
    "errorCode": "DatabaseSource.InvalidConfiguration"
}
```

When testing the DNS for the VPC Endpoint automatically created by Kinesis Firehose, I noticed it fails to connect to the RDS Proxy. Although the IPs resolve via nslookup, I cannot establish a connection using telnet.

My theory is that, for some reason, the VPC Endpoint created by Kinesis Firehose isn’t correctly configuring (within the AWS-managed account) the route to the RDS Proxy or the security rules. Has anyone else faced this issue?

I'm using the sa-east-1 region.

Paulo Henrique Bruce · Accepted Answer

**Resolved**: I discovered that the ELB created to configure the VPC Services cannot have a security group associated with it. I didn’t test with a security group completely open to 0.0.0.0/0 (both inbound and outbound), but previously, I had tried adding inbound rules to the security group for the security group used by the RDS, plus an inbound rule for port 443, as indicated in the AWS documentation. After recreating the ELB without an associated security group, it worked.

Jason · Answer

similar to @Paulo, I found that it was related to security groups on the NLB.  However, my solution was to turn "Enforce inbound rules on PrivateLink traffic" Off via the console.

re:Post Agent · Answer

Based on the information provided, it appears you're experiencing connectivity issues between Kinesis Firehose and your RDS Proxy. This could be due to several factors in your network configuration. Here are some steps you can take to troubleshoot and potentially resolve the issue:

1. Security Group Configuration: Ensure that the security group associated with your RDS Proxy allows inbound traffic from the Kinesis Firehose VPC Endpoint. The security group should permit access on the database port (typically 5432 for PostgreSQL) from the IP range of the Firehose VPC Endpoint.

2. Route Tables: Verify that the route tables for the subnets where your RDS Proxy is located have the correct routes to allow traffic from the Firehose VPC Endpoint. This may involve checking the route to the VPC Endpoint Service.

3. VPC Endpoint Service Configuration: Double-check that your VPC Endpoint Service is correctly configured to allow the Kinesis Firehose service (firehose.amazonaws.com) to connect. Ensure that the principal is correctly set and that there are no additional restrictions preventing the connection.

4. Network ACLs: If you're using Network ACLs, make sure they're not blocking traffic between the Firehose VPC Endpoint and your RDS Proxy.

5. DNS Resolution: Since you mentioned that nslookup works but telnet fails, there might be an issue with the DNS resolution or routing within the VPC. Ensure that DNS hostnames and DNS resolution are enabled for your VPC.

6. RDS Proxy Configuration: Verify that your RDS Proxy is correctly configured to accept connections from the Firehose VPC Endpoint. This includes checking the proxy's security group and ensuring it's in the same VPC as your Aurora cluster.

7. Aurora Cluster Configuration: Confirm that your Aurora cluster is in the correct subnet group and that these subnets are accessible from the Firehose VPC Endpoint.

8. Permissions: Double-check that the database user created for Firehose has the necessary permissions to connect and perform the required operations.

9. VPC Peering: If your RDS Proxy and the Firehose VPC Endpoint are in different VPCs, ensure that VPC peering is correctly set up and that the necessary routes are in place.

10. AWS Support: If you've verified all the above and are still experiencing issues, it may be worth contacting AWS Support. As this is a preview feature, there could be specific configurations or limitations that are not fully documented.

Remember to use tools like VPC Reachability Analyzer or AWS Systems Manager Session Manager to further diagnose network connectivity issues. These can help identify any missing or incorrect network configurations that might be preventing the connection.

Given that you're using a preview feature, it's also possible that there are some limitations or bugs that haven't been fully addressed yet. Keep an eye on AWS documentation updates and consider providing feedback to AWS about your experience with this preview feature.  
**Sources**  
[ECS Service Connectivity Issue with AWS Kinesis | AWS re:Post](https://repost.aws/questions/QU5JLXC5AdRbWYJXsC6yN9qA/ecs-service-connectivity-issue-with-aws-kinesis)  
[Troubleshooting Firehose endpoint reachability - Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/endpoint-troubleshooting.html)  
[Creating an RDS Proxy - Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy-creating.html)

[Solved] Connectivity Issues with Kinesis Firehose Preview Capability and RDS (CDC)

Add your answer

Relevant content