Parallel load support in aws_s3 extension to import data in Aurora PostgreSQL

Question

A customer wants to know whether there is any native mechanism to parallel upload data to Aurora from multiple csv files in S3. Based on the documentation [1] unlike, DMS there is no supported way of doing it. Secondly, is there a way one can modify aws_s3 extension i.e. wondering if have open-sourced it. DMS is another alternative, but wanted to rule out if there I am missing anything specific here.

[1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Migrating.html#USER_PostgreSQL.S3Import

Accepted Answer

As the document you mentioned, aws_s3 uses S3 API to download a file and then uses COPY statement to load data. It supports a single file, not multiple files as input. The customer need to implement to handle multiple files in their application.

aws_s3 is released by RDS/Aurora PostgreSQL team and not seemed to be open-sourced.

[1]: https://www.postgresql.org/docs/10/sql-copy.html

Parallel load support in aws_s3 extension to import data in Aurora PostgreSQL

Relevanter Inhalt