how to manage aws glue jobs?

Question

say, i have some data in s3 bucket and an  aws glue crawler job that reads and creates few aws glue catalog tables. I want to read the data in these tables and push it to some other database like dynamo db or rds, and i can achieve that by setting up a glue job. 
looking for best practices here. do i craete a glue etl job for each catalog table. for example , one etl job per table or can i combine all the etl jobs to update each aws glue catalog table to corresponding table in another database (dynamo or rds) ?

Oleksii Bebych · Answer

You can use many different data storages with Glue

Glue with DynamoDB:

https://aws.amazon.com/blogs/big-data/accelerate-amazon-dynamodb-data-access-in-aws-glue-jobs-using-the-new-aws-glue-dynamodb-elt-connector/

Glue with RDS:

https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.html

how to manage aws glue jobs?

Add your answer

Relevant content