How to sync bidirectionally both s3 bucket and EC2 server?

Question

Hello team,

I have a wordpress application which is deployed in Elastic Beanstalk and static content is delivered by S3 bucket whereas dynamic content has been delivered by EC2 server.

My project has a folder **wp-content/uploads/** in root directory. This **uploads** folder is sync with S3 bucket using the script which is placed in **.ebextension** folder.

Below one is .ebextension script which is going to add in crontab in EC2 server.

```
commands:
  01_set_cron_job:
    command: |
      sudo crontab -l | { cat; echo "*/1 * * * * aws s3 sync --delete --size-only s3://mydemobucket/wp-content/uploads/ /var/app/current/wp-content/uploads/"; } | sudo crontab -
```
 
Whenever new EC2 instance has been launched in my Beanstalk Autoscaling environment, then all the objects are available in S3 bucket is sync with newly launched EC2 instance.

This direction is perfectly working for me.

My problem, that If my developers some images to the **uploads** folder through wordpress application, then how I copy the updated images from EC2 server to S3 bucket. I just tried and little bit confused.

Finally, I can able to sync from S3 to Ec2 server with the help of crontab. This is one direction that is **from S3 to EC2 server**.

I would like to copy the images from **EC2 server to S3 bucket** whenever some changes or new images are updated in EC2 server. This is one direction that is **from EC2 to S3 bucket**.

I need bi-directional sync without any impact. Can any one please help me to achieve this issues?

Answer

[Mountpoint for S3](https://aws.amazon.com/s3/features/mountpoint/) can mount an Amazon S3 bucket as a local file system in your EC2 instance.  It supports basic file system operations, list and read existing files, and create new ones. It cannot modify existing files or delete directories, and it does not support symbolic links or file locking. Mountpoint is ideal for applications that do not need all of the features of a shared file system and POSIX-style permissions but require Amazon S3's elastic throughput to read and write large S3 datasets. Refer to [Working with Mountpoint for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html) for details

If you need a POSIX-style storage, consider EFS. Refer to [How can I mount an Amazon EFS volume to an instance in my Elastic Beanstalk environment?](https://repost.aws/knowledge-center/elastic-beanstalk-mount-efs-volumes) for overview.

Answer

Have you considered using Amazon EFS instead of EBS if you need shared storage accessible from multiple EC2 instances? 
Amazon EFS can be mounted on multiple instances and integrated with services like DataSync for bidirectional sync with S3.
Use AWS DataSync to configure ongoing, automated sync between an EFS file system and S3. AWS DataSync can sync in both directions based on filters you define.

https://docs.aws.amazon.com/efs/latest/ug/trnsfr-data-using-datasync.html

Answer

Running `aws s3 sync --delete --size-only s3://mydemobucket/wp-content/uploads/ /var/app/current/wp-content/uploads/` every minute using **cron** is effectively updating the contents of the `uploads` directory on the EC2 to match the `uploads` prefix in the bucket, and the `--delete` flag means delete anything in the EC2 directory that isn't already in the bucket https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects-sync

When one of your developers uploads a file to the directory on the EC2, the next time the cron job runs (which will be within a minute) this new file will be deleted because of the `--delete` flag.

You may be able to "beat" this by using something like [**incron**](http://inotify.aiken.cz/?section=incron&page=about&lang=en) to kick-off a new sync job any time a change is made on the EC2 and sync that file to the bucket. But that's not 100% guaranteed to happen every time.

As other answers have already said, there are other AWS services that are probably a better fit for your use case.

How to sync bidirectionally both s3 bucket and EC2 server?

Relevant content