3 tier application - slow IO latency between RDS and EC2
Hello,
I've recently re-architected my application, which is written in Ruby on Rails and uses a PostgreSQL 13 database, from a setup where everything was in public networks, as shown in the attached diagram (2x EC2 instances and an RDS instance), to a 3-tier architecture.
I've separated everything into public and two private networks, also as shown in the attached diagram.
Everything is working fine, but I've encountered an issue with the speed of data loading/retrieval from the database.
I've noticed that the IO latency has increased to as much as 25 milliseconds, whereas in the previous setup it was around 2-4 milliseconds.
I've tried upgrading the RDS instance to a more powerful one with a larger CPU, but this hasn't helped.
What could be the reason for this behavior?
- Newest
- Most votes
- Most comments
Hello.
Were EC2 and RDS originally running in the same subnet?
Looking at the configuration diagram, the subnets are separated and communication occurs between AZs.
Communication between AZs will result in a difference in performance compared to communication within the same AZ.
You can perform a simple benchmark test by following the steps in the document below, which may be helpful.
https://repost.aws/knowledge-center/network-throughput-benchmark-linux-ec2
Additionally, IO performance may be improved somewhat by changing the RDS storage settings.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html
@Riku_Kobayashi So the first infrastructure was all in one subdomain. I tried to move the new one to one Azs to eu-cental-1b but it didn't help. There are still EC2 and rds in different subdomains. Can the fact that they are in private networks and EC2 uses a NAT gateway to communicate with the Internet have an impact on latency?
Relevant content
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated 2 months ago
- AWS OFFICIALUpdated a month ago
Since NAT Gateway is not used for communication within the VPC, I don't think it is related to communication latency between EC2 and RDS. I suspect that latency is occurring in communication between subnets in different AZs.