EFS Performance to Ubuntu 14.04.6

Question

Hello AWS Support ,
Greetings for the day and I hope that you are well.
 We have a special situation with a production, We have an instance of ubuntu, which has a fileSystem called s3fs.
Lately it has been failing to dismount, it is worth mentioning that to remount we perform the following actions:
1. sudo su
2. su - git
3.fusermount -u /var/www/resources
4. s3fs -o nonempty -o iam_role='rol_ssm' -o endpoint="us-west-2" -o use_cache=/tmp/cachegit stargroup-resources /var/www/resources

When the FS is unmounted, question marks appear. 
d?????????    ? ?        ?           ?            ?          resources/

Additional I add the data of the instance where the FS is mounted

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

Could you please support us?

Karen T,

Itera´s Cloud Operation Rep

Answer

Hello Karen, Thank you for your message. I understand you are experiencing difficulty with your s3fs mount in Ubuntu 14.04.6 LTS.

To evaluate the issue you reported, I launched an Ubuntu 14.04.6 EC2 instance and manually compiled s3fs. I was then able to mount an S3 bucket to my instance, create new files in it, and successfully unmount and remount the bucket.

In order to reproduce the issue with the question marks appearing in "ls" output, the only action I found that would accomplish this was to forcefully kill the s3fs process without safely unmounting the bucket:  
  
```
# kill -6 21759
# ls -al /mnt/
ls: cannot access /mnt/s3bucket: Transport endpoint is not connected
total 8
drwxr-xr-x  3 root root 4096 Mar 25 17:35 .
drwxr-xr-x 22 root root 4096 Mar 25 17:00 ..
d?????????  ? ?    ?       ?            ? s3bucket
```
  
This indicates that either (1) the bucket is being unmounted improperly, or (2) the s3fs process is being killed unexpectedly.  
  
To avoid issue (1), please make sure that no running processes have open files in the S3 bucket before attempting to unmount it. You can use the "lsof" command to verify this. In the example below, you can see that I am using the "vi" text editor on a file in my S3 bucket:  
  
```
# lsof /mnt/s3bucket/
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
vi      21916 root    4u   REG   0,24    12288    5 /mnt/s3bucket/.hello.swp
```
  
I would need to make sure I quit vi before unmounting my bucket.  
  
As for issue (2) there are many scenarios in which a process can be killed unexpectedly, and I would not be able to speculate without having specific knowledge of your system. However, I can recommend that you use the "auditd" daemon to collect information in the event that the s3fs process is killed. I am including resources below with instructions for installing auditd and configuring a rule to monitor "kill" signals$$1$$$$2$$.  
  
Please let me know if you have any questions.  
  
References:  
$$1$$ https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-track-changes/  
$$2$$ https://www.thegeekdiary.com/how-to-find-which-process-is-killing-mysqld-with-sigkill-or-sigterm-on-linux/

EFS Performance to Ubuntu 14.04.6

Relevant content