write speed through lambda access point
Our goal is to move files sized from 4-50 GB from S3 to EFS using lambda.
EFS is configured as provisioned speed 1000 MiB
successfully mount lambda to the access point. Lambda and EFS in the same VPC, same subnets.
For benchmarking, we are using file with size of 400MByte. Lambda configured to be 2G memory
download 400 Mbyte files from S3 using Java API s3Client.getObject(new GetObjectRequest(
bucket, key), file);
To lambda /tmp directory : 7 sec
To an EFS access point. /mnt/files : 8.8 seconds
Why writing to EFS so slow ~45MB? Is it because downloading from S3 has a slow pipeline?
Please advise the right way to do it. We would like to stay with Lambda instead of using ECS (EC2) instance in this case.
if lambda is configured to be 256M Byte of memory, the speed de-grade significantly
download 400 Mbyte from S3 to
/tmp : 30 sec
/mnt/g : 44.7 sec.
Hi - thanks for trying out our recent EFS+Lambda integration! EFS should be able to achieve a higher write speed to a large file for a test like this. It looks like the performance of downloading to /tmp and /mnt/files is comparable in both of your cases, so it seems like the download from S3 is the limiting factor. When you give your Lambda function 3 GB/s of memory does the performance improve? Similarly, when you write a file from /tmp to /mnt/files is the performance better? Both of these will help narrow down the bottleneck.
Thank you for the answers. They are in-line with what we are thinking and we are testing with lambda which is setup with 3008M byte RAM.
Any official AWS document clearly stated the speed of moving large file from S3 to either lambda /tmp or EFS access point? We have read some online experience where people saying it is 9-25Mbyte/sec but we would love to see some AWS document for that.
Relevant questions
Mount EFS in Workspaces Stuck in Read-Only
asked 6 months agoEFS - files not found sometimes and randomly
asked a month agoBest way to expose files from EFS over HTTP(S)?
asked 6 months agoEFS price based on giga bytes:
asked 2 years agoUnzipping files from S3 bucket
asked 5 months agoOld EFS Access Points work but cannot mount a new one?
asked 2 years agoHow to save a file to a tmp/temp folder in lambda?
asked 21 days agowrite speed through lambda access point
asked 2 years agoClearing files on EFS from Lambda after DataSync Operation from S3
asked a month agoHow do I save data files to EFS using a Node.js application?
asked 3 years ago