write speed through lambda access point

Question

Our goal is to move files sized from 4-50 GB from S3 to EFS using lambda.  
  
EFS is configured as provisioned speed 1000 MiB  
successfully mount lambda to the access point.  Lambda and EFS in the same VPC, same subnets.  
  
For benchmarking,  we are using file with size of 400MByte. Lambda configured to be 2G memory  
download 400 Mbyte files from S3 using Java API s3Client.getObject(new GetObjectRequest(  
                bucket, key), file);  
  
To lambda /tmp directory : 7 sec  
To an EFS access point. /mnt/files : 8.8 seconds  
  
Why writing to EFS so slow ~45MB?  Is it because downloading from S3 has a slow pipeline?  
Please advise the right way to do it.  We would like to stay with Lambda instead of using ECS (EC2) instance in this case.  
  
if lambda is configured to be 256M Byte of memory, the speed de-grade significantly  
download 400 Mbyte from S3 to  
/tmp : 30 sec  
/mnt/g : 44.7 sec.

Answer

Hi - thanks for trying out our recent EFS+Lambda integration! EFS should be able to achieve a higher write speed to a large file for a test like this. It looks like the performance of downloading to /tmp and /mnt/files is comparable in both of your cases, so it seems like the download from S3 is the limiting factor. When you give your Lambda function 3 GB/s of memory does the performance improve? Similarly, when you write a file from /tmp to /mnt/files is the performance better? Both of these will help narrow down the bottleneck.

Answer

Thank you for the answers.    They are in-line with what we are thinking and we are testing with lambda which is setup with 3008M byte RAM.  
  
Any official AWS document clearly stated the speed of moving large file from S3 to either lambda /tmp or EFS access point?  We have read some online experience where people saying it is 9-25Mbyte/sec but we would love to see some AWS document for that.

write speed through lambda access point

相關內容