How can I throttle the write operation from a glue import job to RDS mysql

0

I want to import 20 million records from a csv to RDS using glue. The glue job will run once a quarter only. I want to be able to throttle the write operations from glue to RDS to limit the IOPS to reduce costs. How can this be done? Is there a way to limit the number of concurrent connections from glue to RDS or any configuration that can help reduce the IOPS. One way to accomplish this is to split the csv into smaller chunks and then process smaller files - but it seems like an overkill just to reduce the IOPS. Update - RDS is cross account and a shared resource so I cannot change any RDS configs

1 Answer
0

Normally you don't limit IOPS since the DB will do that "throttling" by taking longer, the main impact in the number of connections in parallel, which you can control by controlling the number of cores in the cluster or the number of partitions (using repartition or coalesce) in the data (whichever is lower).

profile pictureAWS
EXPERT
answered 6 months ago
  • Sorry I should have mentioned, RDS is cross account and a shared database so I cannot change RDS config

  • That's not what I suggested (that would throttle all users equally), all I talked about is done in the Glue code

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions