Using Lambda to run s3 export that takes longer than 15 mins

Question

Hi , 
I have a lambda that calls s3 export on aurora postgres tables . Couple of tables take longer than 15 mins . This causes the s3 export to fail in some cases . 
Is there a way I can just use lambda to trigger the s3 export and then let aurora take care of completing the query . I can use a step function and another lambda to check if there is any s3 export still running and take the status of s3 export from their using wait and a loop . 
I wanted to ask how can i let the first lambda just trigger the s3 export and not fail when the response takes longer than 15 minutes for longer running exports . 
Thanks 
Pradeep Singh

Answer

Unfortunately, Lambda cannot run for more than 15 minutes.  
https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html#configuration-timeout-console

So I think the only way to handle this is to use the Step function as you are doing.

If a process takes more than 15 minutes to execute Lambda, it may be better to use ECS or AWS Batch.

Answer

Hi, another way is to use a container image as a Fargate service. The Lambda + Step may not work as the hard 15 min limit may interrupt the synchronous SDK call for S3 export.

We had a similar case and went to Fargate for long calls to other AWS services.

Answer

Yeah . We wanted to use Fargate but that's kind of not allowed at this moment . We will have to wait for approvals and that depends on another project that is running currently . Is there a way to make the SDK call for S3 export asynchronous

Answer

Can I use StartExportTask activity in Step function . Can it be used to provide custom queries for export to s3

Using Lambda to run s3 export that takes longer than 15 mins

Relevanter Inhalt