Looking for a tool for faster aws glue test iterations

0

I'm building several glue jobs for a full database migration to a new database with different schema (but similar business logic).

Glue is a powerful tool, however it's painful the time I need for each iteration to test something I did. If I want to run the job it averages between 2 mins and 6 minutes (based on the job and the amount of relations and data).

I found out I can use the data preview session. This made it faster to test changes, however we're still talking about 30 seconds for each refresh of the session (and more than a mintue for first run) + the data preview doesn't attempt to insert into the destination DB so some issues (unique constraint problems, etc..) won't appear until I do the actual job run.

Is there a best practice or tool that I can use to mimick the glue jobs but have them react in a much faster way?

p.s. I tried to use bigger and more workers in the job but still we're talking about >1 minute durations even with that.

質問済み 8ヶ月前162ビュー
2回答
0

The fastest way to develop on a real environment (not a docker or local) is using Glue Notebooks which are based on Interactive sessions.
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#develop-using-interactive-sessions

profile pictureAWS
エキスパート
回答済み 8ヶ月前
  • I havent' tried the glue notebooks. Will try to check them out. Thanks

0

Hello,

As per your usecase where you want to avoid running the complete glue job to test the code you can utilize glue interactive sessions [+] https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions-gs-notebook.html

Glue interactive sessions help our customers to test individual snippets of code and debug the code in real time.

AWS
Sahil_S
回答済み 8ヶ月前
  • Sessions are better but still take time and don't test the actual insertion. But they're an improvement over running the job all together for sure. Thanks

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン