By using AWS re:Post, you agree to the Terms of Use

write speed through lambda access point

0

Our goal is to move files sized from 4-50 GB from S3 to EFS using lambda.

EFS is configured as provisioned speed 1000 MiB
successfully mount lambda to the access point. Lambda and EFS in the same VPC, same subnets.

For benchmarking, we are using file with size of 400MByte. Lambda configured to be 2G memory
download 400 Mbyte files from S3 using Java API s3Client.getObject(new GetObjectRequest(
bucket, key), file);

To lambda /tmp directory : 7 sec
To an EFS access point. /mnt/files : 8.8 seconds

Why writing to EFS so slow ~45MB? Is it because downloading from S3 has a slow pipeline?
Please advise the right way to do it. We would like to stay with Lambda instead of using ECS (EC2) instance in this case.

if lambda is configured to be 256M Byte of memory, the speed de-grade significantly
download 400 Mbyte from S3 to
/tmp : 30 sec
/mnt/g : 44.7 sec.

asked 2 years ago53 views
2 Answers
0

Hi - thanks for trying out our recent EFS+Lambda integration! EFS should be able to achieve a higher write speed to a large file for a test like this. It looks like the performance of downloading to /tmp and /mnt/files is comparable in both of your cases, so it seems like the download from S3 is the limiting factor. When you give your Lambda function 3 GB/s of memory does the performance improve? Similarly, when you write a file from /tmp to /mnt/files is the performance better? Both of these will help narrow down the bottleneck.

answered 2 years ago
0

Thank you for the answers. They are in-line with what we are thinking and we are testing with lambda which is setup with 3008M byte RAM.

Any official AWS document clearly stated the speed of moving large file from S3 to either lambda /tmp or EFS access point? We have read some online experience where people saying it is 9-25Mbyte/sec but we would love to see some AWS document for that.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions