By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

SSH not able to authenticate when /home/<user>/ directory is mounted to an EFS directory

0

I am setting up new RHEL 8 EC2 instances for my team to use to log into and do work directly on those machines. My goal is to have common /home/<user>/ directories across all the machines, so I was happy when I found this article from the EFS User Guide about mapping EFS directories to /home/* directories via a standard mount process: https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions-per-user-subdirs.html. I followed the instructions exactly and it worked in that I could access common home data across multiple machines.

The problem comes when I want to login to my account on any one of the instances. I issue an ssh command of the following form:

ssh -i <my PEM file> <username>@<instance DNS>

and it returns the following message:

<username>@<instance DNS>: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I have confirmed that the ~/.ssh/authorized_keys file contains the public key generated from <my PEM file>. In fact, I didn't mount an ec2-user account directory as a safety precaution so that I could guarantee being able to have access in case there was a problem. So, I am able to log into the instances as ec2-user and then confirm the contents of my /home/<username>/.ssh directory. I use the same PEM file for both ec2-user and <username>, so my .ssh/authorized_keys files are common between both accounts. The only difference is that the ec2-user is not physically using the EFS mount, but the <username> account is.

Taking this experiment a step further, I used umount to remove my <username> home directory mapping to EFS. At this point, I copied the .ssh/authorized_keys file from the same user's home directory on the EFS drive to my root drive /home/<username>/.ssh directory (to ensure it was exactly the same). After these steps, I was able to ssh directly to my <username> account. To confirm one more level, I remounted the EFS home directory and confirmed the ssh to <username> no longer worked (which was the case).

This debug is quite conclusive as to the problem. As I have another global path to this EFS home directory through /mnt/<efs_mount>/<username>/, I added that path to the .ssh/authorized_keys file as a second item in the AuthorizedKeysFile variable in my /etc/ssh/sshd_config file then executed systemctl daemon-reload and systemctl restart sshd. But when I tried the ssh to the <username> account, it still gave me the same publickey error.

Any solution or guidance would be very appreciated.

Topics
StorageCompute
Tags
Amazon Elastic File SystemAmazon EC2
Language
English
Darren
asked 2 months ago171 views
2 Answers
1
Accepted Answer

Going back to the scenario where username has a home directory on the local disk, and the EFS filesystem is mounted on /mnt, and you can login as username without any problems.

When logged in as username are you able to cd into the home directory on EFS (which isn't the "real" home directory of course) as that user? If you can, confirm the file ownership within the directory is correct, and in particular that it isn't nobody or nfsnobody.

profile picture
EXPERT
Steve_M
answered 2 months ago
profile picture
EXPERT
Oleksii Bebych
reviewed a month ago
  • Darren
    2 months ago

    Hopefully this reply is in the right place. Answering your questions takes more than 600 chars, so chopping it into parts.

    You are correct that the EFS mount point is at /home/<username> (matching the EFS User Guide). I created the EFS directory matching that as a child of the root on the EFS drive. So, you are correct that they are mapped such that they will satisfy the /etc/passwd file. Only the default instance account (ec2-user) has its home dir on the local EBS root drive. Also, I mounted the EFS root to /mnt/efs_home/ on the local drive. This /mnt mount point is owned by root.

  • Darren
    2 months ago

    Here is the /etc/fstab entry for the /home/<username> mount (with personal info removed):

      <efs ID>:/<username> /home/<username> efs nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,nor        esvport,_netdev 2 0
  • Darren
    2 months ago

    The output of "sudo mount | grep nfs" (run as ec2-users to sudo, personal info removed):

    <efs ID>:/ on /mnt/efs_home type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.31.0.214,local_lock=none,addr=172.31.11.186) <ebs ID>:/ on /tools type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.31.0.214,local_lock=none,addr=172.31.14.207,_netdev) <efs ID>:/<user> on /home/<user> type nfs4 (<to save space: same efs param string>)

  • Darren
    2 months ago

    The "setsebool -P use_nfs_home_dirs 1" was the magic that did the trick! I am now able to ssh directly to my <username> account. It would be great to add this tidbit to the EFS User Guide section that this exercise started from to avoid this loop for any others. Thanks for the great help to diagnose this issue and provide a solution. Thanks!

  • Steve_M EXPERT
    2 months ago

    You're welcome.

    There's a feedback link at the top-right of the documentation page which would be the best place to make that suggestion. While Amazon Linux will have SELinux set to permissive by default (and so this issue will never be seen), as you have found other flavours of Linux have set to enforcing by default.

0

Hi Steve,

Thank you for the good point you raised. I should have mentioned that in my original question. Also, I am "answering here" because I don't see a way to reply directly to your answer with an answer to your questions. I'm new to both AWS and re:Post, so please help me understand the protocol for interaction here if I am doing this wrong.

Now to answer your question... The directory contents, file ownership, group ownership, and permissions are exactly the same between the view that I see in the /home/<username>/ directory and the /mnt/.../<username/ directory. I went back just now to confirm this is the case again. I will also add that I confirmed that the /home/<username/ content were clearly different (in files other than the authentication related files) when I unmounted that directory in my debugging. I purposefully had unique contents to make sure I recognized which one is which.

I can't accept the answer yet, but thanks for the attention so far.

Darren
answered 2 months ago
  • Steve_M EXPERT
    2 months ago

    No worries, we were all newbies here once :-) Reply to an answer by clicking Comment under the reply.

    Reading through your original question again, you say

    The only difference is that the ec2-user is not physically using the EFS mount, but the <username> account is.

    Assuming the ec2-user's home directory is /home/ec2-user and is on the local disk (because it works), at this point where is EFS mounted?

    It can't be mounted at /home because if that happened /home/ec2-user would have disappeared when EFS was mounted on top of it. It would have to be directly at /home/<username> like in the link that you posted in your question that's the only way this would work (without going down the road of bind mounts or loopback mounts).

    If it is, make sure that the ownership of the top-level of the mountpoint is <username> and not <root> (which it would probably be the default).

    if EFS is mounted anywhere else then the homedir for <username> won't align with the entry in /etc/passwd.

    If it still doesn't work, could you post the entry in /etc/fstab for EFS, as well as the output of sudo mount | grep nfs?

  • Steve_M EXPERT
    2 months ago

    One other thing, what's the OS you're running, and is SELinux enforcing?

  • Darren
    2 months ago

    $cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 (Ootpa)

    $sudo sestatus command SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33

  • Steve_M EXPERT
    2 months ago

    What's the output of getsebool use_nfs_home_dirs

    it looks like this needs to be set to on to use NFS home directories https://linux.die.net/man/8/nfs_selinux

    If you want to use a remote NFS server for the home directories on this machine, you must set the use_nfs_home_dirs boolean:
setsebool -P use_nfs_home_dirs 1

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content