Design suggestions

Hello All

I am expanding the scope of the project, and wanted some suggestions/comments on if the right tech stack is being used. (Job - Pulling Leads and Activities from Marketo). Each job has a different query.

Scope: We need 2 jobs to be run daily, however here is the catch. Each job should be created and queued first. Once done, we can poll to see if the job on Marketo side is completed and the file is downloaded. The file download can take more than 15 mins. THe file an be downloaded using the Job Id, using the 2 jobs that were created earlier.

Current/Implemented Solution: I started with solving for Leads only and here is the stack that was worked out. The Event will be triggered, on a daily basis using event bridge. The task that is to be triggered is a step function. The Sfn, first calls a Lambda to create job, waits for 1 min, another lambda to queue the job, then wait for 10 mins and 3rd lambda to check status of file. If not ready, wait for 10 and poll again for file status(this is a loop with choice to keep checking file status). Once file is ready, call a container(fargate/ECS) and pass the Job Id as containerOverride to the container. Run the job on container to download the file and upload the file to S3.

Incorporate pulling activities into the above flow: Since the queuing and Polling(for file status) lamba are similar, and the first lambda(creating the job) is different, I though of creating a parallel task where each branch does create, queue, poll and download the file(using the implemented solution, so 2 lambdas for job create and reuse the 2nd and 3rd lambdas). Once the complete chain(one for Leads and one for activities) is done, have a consolidation stage where the output from each container output is collected and an SNS message of job completion is send.

I am looking forward to your suggestions to see if the suggested above workflow is how it should be done or is there any other technology that I should use.

Design Constraints: I need to create and queue all the jobs first before starting the downloading since Marketo has a limit of 500mb for file download. Hence I the need to create and queue all the jobs first , and then only start the job to download of files.

Thanks for your suggestions.

Regards Tanmay