Process large data file ( up to 10 GB) and save to RDS

Question

Hi 
I have an architecture like below  
user upload file -> S3 -> lambda trigger glue job -> glue job pull the file, read content, and save to a record in a table in Aurora Postgres  
Everything is good with a small file, but when file size increases (up to 10Gb), I think the architecture can not fit anymore.
I researched and found that Postgres has something like an external table that can store much more data than a regular table.
I also think about switching to a NoSQL database like Dynamo or MongoDB
I have some questions:
- Aurora Postgres is good for storing and searching large content?
- If Aurora Posgres is not good then which NoSQL database fits this scenario?

Answer

Hello.

> Aurora Postgres is good for storing and searching large content?

PostgreSQL is a relational database, so it is suitable for performing complex queries such as table joins.  
Therefore, I think it is suitable for performing complex updates and searches.

> If Aurora Posgres is not good then which NoSQL database fits this scenario?

NoSQL such as DynamoDB is faster than relational databases, which are faster to read and write.  
However, it is not suitable for systems that require complex data updates.

We recommend that you check the following documentation to select the appropriate database for your solution.  
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SQLtoNoSQL.WhyDynamoDB.html

Process large data file ( up to 10 GB) and save to RDS

相关内容