Compare two table using AWS service

Question

Hello All, 
I have use case which I am trying to implement using serverless services of AWS . I have source table S1 which has some data .
We have target table T1 having same columns (with some extra columns ) .Both tables shares same Primary key column.Both table are in same schema in database .Now what is best way to compare two tables for CDC based on Primary key and perform upsert in T1 ?

Also After upsert of data in T1 let say we have found 1 million record got updated . Now we want pass those records to pass through some engine which performs some calculation  and update dew extra columns of all those rows . Each row processing can be independent . So how can I achieve this ?

I cannot use lambda because it has limitation of 15 mins and tables are quite big.

Answer

This is a fairly standard computer science question - there are many algorithms out there that are available to compare two data sets - I'm sure you've researched this so my answer is more about which service to use.

Given that the Lambda maximum runtime is a limitation, you might consider using Fargate.

Answer

This is  more like ETL use case. You  could use AWS GLUE services. Glue can read and compare, which is fairly a query, and then it can process or populate additional column based on business requirement.  Glue supports multiple framework and programming languages.  Glue can read/write from multiple data sources including S3, rdbms, redshift etc.
https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html

Compare two table using AWS service

Relevant content