As you say, EFS is not ideal for this use case. EFS provides low-latency access to large volumes of information, but only through an otherwise-opaque NFS file system interface. It does not provide an HTTP API for its contents, nor does it produce lifecycle events for files in it.
I'd say your best bet is #2, replicating to S3. While it requires more modifications to your existing code, it has fewer moving parts overall and should be lower maintenance. You'll have to hook into your existing thumbnail generation process, but there are a few options for that. Without modifying your thumbnail generation Lambda function, probably the easiest is to add a Lambda destination onto the thumbnail generation function. You'll create a new Lambda function, which will get the input and output of the thumbnail generation function, and can pull the thumbnail from EFS and put it into Lambda.
AWS DataSync This becomes your replication mechanism.
In using a managed service to simplify the process of transferring data from an EFS filesystem to an S3 bucket. You can also schedule periodic replication of an EFS file system to an S3 bucket. https://aws.amazon.com/premiumsupport/knowledge-center/datasync-transfer-efs-s3/
Note/Cost Optimization - VPC Endpoints for DataSync use AWS PrivateLink, when you use DataSync in a VPC the agent can communicate directly without the need to cross public internet. https://docs.aws.amazon.com/datasync/latest/userguide/datasync-in-vpc.html
- asked a year ago
- asked 2 years ago
- Accepted Answerasked a year ago
- Accepted Answerasked 4 years ago
- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 10 months ago
- AWS OFFICIALUpdated 8 months ago
- EXPERTpublished 25 days ago