Parallel load support in aws_s3 extension to import data in Aurora PostgreSQL
A customer wants to know whether there is any native mechanism to parallel upload data to Aurora from multiple csv files in S3. Based on the documentation 1 unlike, DMS there is no supported way of doing it. Secondly, is there a way one can modify aws_s3 extension i.e. wondering if have open-sourced it. DMS is another alternative, but wanted to rule out if there I am missing anything specific here.
1 https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Migrating.html#USER_PostgreSQL.S3Import
As the document you mentioned, aws_s3 uses S3 API to download a file and then uses COPY statement to load data. It supports a single file, not multiple files as input. The customer need to implement to handle multiple files in their application.
aws_s3 is released by RDS/Aurora PostgreSQL team and not seemed to be open-sourced.
Relevant questions
Changes to connection handling with Aurora Postgres?
Accepted Answerasked 3 years agoHow to import postgresql or nosql datasets in Amazon Sagemaker?
asked 5 months agoParallel load support in aws_s3 extension to import data in Aurora PostgreSQL
Accepted AnswerPostGIS and Aurora compatibility: experience feedback
Accepted AnswerAurora PostgreSQL aws_s3.query_export_to_s3 compression
asked a month agoHow to ensure replica is in sync between AWS RDS PSQL primary DB and Aurora RDS PSQL replica cluster?
asked a day agoQuicksight export more than 1 million rows data to multiple csv
asked 3 months agoMigration from Aurora PostgreSQL back to regular RDS
Accepted Answerasked 3 years agoMajor Version Upgrade Aurora PostgreSQL 11
asked 2 years agoDoes Neptune execute bulk loader jobs in parallel or serially?
asked 2 years ago