By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Polling using Lambda

0

Hello All

I am querying Marketo bulk API. This is how you get data from Marketo:

  1. Set up a job
  2. Enque the job
  3. Keep checking Job status for Completed
  4. Once completed, download the file/data

I have question on how to accomplish step #3. Here is what my thought process is:

  • Set up a lambda to fire once a day
  • Complete step#2 and set the job Id to SQS with a delay of 10 mins
  • fire a Lambda once message is visible to Poll the api for status check

The question is what is a good architecture to proceed post this step if the file is ready to be consumed or if the file is not ready and another poll needs to be done.

If File is ready - Should I fire another lambda(with more memory and timeout) to consume the data and process it?

File not Ready - Is the best way to delete the message and post it back to Queue? Or is there another architecture that is simple and better? Can I leverage Step function to make my life easy ?

Or is there s completely different architecture that I should leverage ?

I do need a Python Package to query Marketo.

Looking forward to your thoughts.

Regards Tanmay

Topics
ServerlessComputeApplication IntegrationDatabase
Tags
AWS LambdaAmazon Simple Queue ServiceExtract Transform & Load Data
Language
English
tkansara
asked 2 years ago1461 views
2 Answers
3
Accepted Answer

Using AWS Step Functions is the way to go when you're trying to setup polling without making it too complex.

The workflow using SQS isn't unheard of. I have personally used it too. But this is a problem that Step Functions can solve in a better way.

Here's an example on polling using AWS Step Functions

AWS
Pronoy_C
answered 2 years ago
  • tkansara
    2 years ago

    Thank you, I was looking at the same example. However I am trying to do it without the Batch Job. Ty

    Also what I understand is you are suggesting to implement the complete flow without using the SQS, so move the #2 lambda in my case as part of step function, and put a default timer of 10 mins, and the proceed with the polling checks?

  • Marco
    2 years ago

    That is only an example. You can also use a Lambda function as the first Element of the execution. With Step Functions you can orchestrate your workflow, and you can interact with over 200 services including Lambda. https://docs.aws.amazon.com/step-functions/latest/dg/connect-supported-services.html

    So i also strongly recommend to use Step Functions for this use case.

0

Not sure if it is applicable, but, did you try the integration of Amazon AppFlow and Marketo? If it meets your requirements you can have a fully automated solution. https://docs.aws.amazon.com/appflow/latest/userguide/marketo.html

javier
answered 2 years ago
  • tkansara
    2 years ago

    Hi The reason I did not choose Appflow was since it does not support Assets. Also I need something that can enque jobs first and then pull data, since running a fill job will cause the 500MB exceeded error.

    My understanding is that if we can queue the 2 jobs first, then it does not matter If I exceed the limit, I can still download the files. Thanks

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content