Aurora Limitless: New shard from limitless_split_shard() causes pg_hba.conf rejects connection and foreign server not found errors in router logs

Question

Hello,

I’m testing Aurora PostgreSQL 16.8 Limitless in a fresh new cluster and shard group.
Every time I run `SELECT rds_aurora.limitless_split_shard('');` the new shard is created successfully, but I consistently see the following issues in the logs.

**Observed behavior**
1. DistributedTransactionRouter logs
```
ERROR: foreign server not found corresponding to id X
LOG: background worker "limitless-gather-node-stats" (PID 8640) exited with exit code 1
```
2. DataAccessShard logs (initial after shard creation) show repeated restarts and config reloads:
* Missing pg_tblspc directory warnings
* couldn't get node topology (missing /opt/rds/topology/kermitTopology.json)
* Multiple invalid configuration parameter name "rds.*" errors
* invalid value for parameter "rds_aurora.limitless_single_shard_query_optimization": "1"
3. Shortly after shard creation, I get pg_hba.conf rejects connection errors:
```
FATAL: pg_hba.conf rejects connection for host "172.31.x.x", user "rds_aurora_limitless_auth_admin", database "postgres_limitless", SSL encryption
```
The IP addresses shown at logs are not in my VPC CIDR -- they appear to be AWS internal addresses.

4. Pattern noticed:
* When I add a new shard, the above errors for the previous shard stop.
* This happens every time, consistently, in a fresh cluster.

**Steps to reproduce**
1. Create a new Aurora PostgreSQL 16.8 Limitless cluster & shard group.
2. Add an initial shard.
3. Call rds_aurora.limitless_split_shard('').
4. Check DataAccessShard and DistributedTransactionRouter log streams for the above errors.

**Questions**
* Is this a known issue with Aurora Limitless shard provisioning or foreign server mapping after limitless_split_shard()?
* Could the AWS internal IPs in pg_hba.conf rejects connection be expected during internal orchestration, or is this a misconfiguration?

Any insight from the Aurora team or other Limitless users would be appreciated.

YUTAKA_H · Answer

Thank you for reporting this detailed observation about Aurora Limitless Database shard behavior. While the errors you're seeing might be related to the internal orchestration process, I recommend opening an AWS Support case for this issue. Our product team can then properly investigate the logs, analyze the specific patterns, and provide appropriate guidance for your Aurora Limitless Database deployment.

I hope this might help.

re:Post Agent · Answer

This behavior appears to be related to the normal provisioning and configuration process for new shards in Aurora PostgreSQL Limitless Database.

Aurora PostgreSQL Limitless Database has a complex architecture consisting of routers and shards that work together in a DB shard group. When you execute `rds_aurora.limitless_split_shard()`, the system creates a new shard, which requires various configuration steps and internal connections to be established.

The errors you're seeing are likely part of the internal orchestration process:

1. The "foreign server not found" errors in the DistributedTransactionRouter logs occur because the router is trying to establish connections to the newly created shard before all metadata is fully propagated.

2. The pg_hba.conf connection rejections from internal AWS IP addresses (172.31.x.x) are likely temporary authentication issues during the shard initialization process. These IPs are part of AWS's internal network used for communication between routers and shards within the DB shard group.

3. The configuration reloads and parameter errors in the DataAccessShard logs are part of the initialization process as the new shard is being configured with the appropriate Aurora Limitless settings.

It's expected behavior that when you add another new shard, the errors for the previous shard stop, as by that time the previous shard has completed its initialization and is fully integrated into the DB shard group.

The individual shards and routers in the DB shard group aren't directly visible or manageable in your AWS account - they're managed by the Aurora service. The errors you're seeing are likely part of the normal provisioning workflow and should resolve themselves as the system stabilizes.

If these errors persist and cause actual functionality issues with your database operations (rather than just appearing in logs), you may want to contact AWS Support for further investigation.  
**Sources**  
[Aurora PostgreSQL Limitless Database architecture - Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/limitless-architecture.html)  
[Benchmarking Amazon Aurora Limitless with pgbench | AWS Database Blog](https://aws.amazon.com/blogs/database/benchmarking-amazon-aurora-limitless-with-pgbench/)

Aurora Limitless: New shard from limitless_split_shard() causes pg_hba.conf rejects connection and foreign server not found errors in router logs

Add your answer

Relevant content