- Newest
- Most votes
- Most comments
Hi,
If your use case requires copying the file once it is created at source location, a common solution is to configure an S3 Event Notification and invoque a Lambda function to execute your logic (in this case, copy the file to another path).
However, if the number of files is huge, it is better to configure the notification from S3 to SQS, and then process the events in batch through the lambda function.
Note: As described in the official documentation, event notifications are delivered in seconds but can sometimes take a minute or longer.
Have you looked at S3 replication - https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html? It removes the need for any customised code, and objects can be replicated within 15 minutes of being created in the source bucket.
Please take a look in this article: https://repost.aws/knowledge-center/s3-large-transfer-between-buckets
Another option mentioned there is S3DistCp tool. It is suitable for periodic transfers. For the "large number of small files" use case, the --srcPrefixesFile option could be useful since it allows to provide a list of prefixes (folders) to copy, so the tool won't have to ls everything in the source bucket.
S3DistCp tool runs on EMR service so you have to consider its cost, but it may be quite efficient (i.e. quick) compared to alternative solutions.
Relevant content
- Accepted Answerasked 10 months ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 6 months ago
- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated a year ago
Yes this can be used but as I mentioned earlier, source file will be huge and huge in number which can throttle the Lambda, it may find concurrency limit to execute like problems.
Totally agree! I just edited the answer but you had answered before.
HI Mikel, thanks for this approach, where will be that batch option available? SQS or Lambda? So that Lambda will not be triggered for every small files with instance rather, Lambda should be triggered batch like may be 100 files.
Hi,
The batch options are configured at the lambda trigger, using
Batch size, andBatch windowproperties. The next page shows how to do it step by step.The
Batch sizedefines the number of records to send to the function in each batch, and theBatch windowthe maximum amount of time to gather records before invoking the function, so you should adjust them based on your use case.Your Lambda function will be executed when one of these two values is reached. That is, when the configured maximum batch size is reached or when the batch window expires.