How can I throttle the write operation from a glue import job to RDS mysql

Question

I want to import 20 million records from a csv to RDS using glue. The glue job will run once a quarter only. I want to be able to throttle the write operations from glue to RDS to limit the IOPS to reduce costs. How can this be done?
Is there a way to limit the number of concurrent connections from glue to RDS or any configuration that can help reduce the IOPS.
One way to accomplish this is to split the csv into smaller chunks and then process smaller files  - but it seems like an overkill just to reduce the IOPS.
Update - RDS is cross account and a shared resource so I cannot change any RDS configs

Answer

Normally you don't limit IOPS since the DB will do that "throttling" by taking longer, the main impact in the number of connections in parallel, which you can control by controlling the number of cores in the cluster or the number of partitions (using repartition or coalesce) in the data (whichever is lower).

How can I throttle the write operation from a glue import job to RDS mysql

相关内容