By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Trigger stored procedure upon glue job succe

0

Hi Team,

I have a web service triggered by lambda every time my glue job succeeded.

This web service runs a bulk raw SQL query against MySQL RDS DB.

I want to replace the web service with a stored procedure.

This stored procedure should be triggered every time my AWS glue job succeeds in order to do some insert/update...

what would be the best practice/way to architecture this?
where should my stored procedure live?

run AWS glue job => job succeded? => trigger my stored procedure to request against MYSQL RDS DB.

Topics
AnalyticsServerlessComputeMigration & ModernizationDatabase
Tags
AnalyticsAWS LambdaAmazon Relational Database ServiceAWS GlueExtract Transform & Load Data
Language
English
Jess
asked 2 years ago519 views
1 Answer
0

Hi there, there's a few ways you can do this and the following are some of the considerations factors:

  • Upon job succeed, do you need to trigger any other workflows apart from the stored procedure
  • Duration of stored procedure
  • Frequency of the job running

Assuming you aren't triggering any other workflows and the duration + frequency of the jobs isn't high, you can stick to the lambda approach since it's serverless and can scale together with your glue job . Stored procedure wise, you can store them directly within the MySQL RDS DB by making some config changes to your DB - you can follow this guide here .

If your considerations are different and you might require to trigger additional workflows etc, high duration etc. You can consider the use of event bridge(enabling fan out architecture upon glue job completion) and other AWS services (i.e code build) for your solution.

Here is a technical guide that demonstrates how to use the combination of CodeBuild and EventBridge rules to schedule and run functions or stored procedures on an RDS for PostgreSQL database instance.

Hope it helps. Cheers.

AWS
AWS-Vincent
answered 2 years ago
  • Jess
    2 years ago

    Thanks a lot for your answer; here are some facts about my job :

    • Upon job success, do you need to trigger any other workflows apart from the stored procedure => no only stored procedures
    • Duration of stored procedure => is about upsert requests against 15 tables average row by table 12k rows
    • Frequency of the job running => the glue job might run twice a day and can take about 14 min

    Which scenario is better for these facts from the above? if lambda what would be the responsibility of lambda here? only connect to my rds DB and invoke the stored procedure?

    if lambda? => if lambda invokes my stored procedure and may stored procedure take say 10 min to finish in this case lambda remain blocked

    for 10 min until it gets a response back from the stored procedure ? or it invokes the stored procedure and stops?

    Thanks a lot.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content