Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Data Warehouse or Data Lake and Suitable ETL Tool for a Startup on AWS

0

Hello,

I work as a data engineer and business intelligence specialist for a fintech startup. We've entered into a new agreement with a supplier to provide a technological solution for managing their finances.

They have tasked me with organizing the data aspect of this solution. Our supplier has provided us access to an RDS where they store various tables that generate information about customers, etc. We will be allowed to extract data from these tables.

Table size: Initially, there will be a few tables. The largest tables have about a million rows, growing by 10 to 30 thousand rows monthly.

Update frequency: Updates every 2 hours are acceptable. However, we need real-time updates for one specific table.

I have 2 questions about this:

To be cost-efficient while managing the information: What do you think is the best solution for my company that relies on AWS? Options like Redshift serverless, Redshift Standard, S3 + Athena, Snowflake, PostgreSQL (I know this is OLTP row-oriented but maybe it's cheaper), or something else?

Would Glue be the most economical solution for ETL, or would Lambda + EventBridge or another approach be better?

I appreciate any help you can provide

Topics
AnalyticsServerlessCompute
Tags
AWS GlueAmazon RedshiftAmazon Redshift ServerlessAWS Lambda
Language
English
asked 2 years ago484 views
1 Answer
0

What’s their rds, mysql or Postgres? If it’s MySQL u can use redshift with zeroetl. If postgres there’s plenty of off the shelf stuff (dms, stitch, etlworks), or you can roll your own with scheduled snapshot exports and spectrum.

Ur aws, be aws, **** snowflake. Unless u have a specific use case for it over redshift, use redshift.

Don’t go Postgres u’ll be indexing ur reporting queries to wazoo and the first time they want a row count on a live filtered grid in their bi ui ur screwed.

Athena has no front end, the best front end is basically redshift spectrum :)

Careful with serverless costs could add up, but at your usage levels a tiny 8/16 round might be good enough. Also apparently there’s gonna be a new smaller ra3 node coming out soon email aserm@amazon.com for early access.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content