- Newest
- Most votes
- Most comments
Hi ProlucidDavid,
I assume you are working in the default directory /home/<cluster_user>.
If this is the case, we always share /home from head node to all compute nodes via NFS, so essentially you are accessing the same /home/<cluster_user>/file.txt from both the head node and compute node.
We also share a number of directories via NFS from head node to compute node, depending on your cluster configuration. You can check which directories are shared by checking /etc/exports on the head node.
If you are looking for other filesystem options, we also support other types of shared filesystem such as EFS and FSx for Lustre.
Hope that helps! Please let us know if you have any additional question
Edited by: AWS-Rex on Dec 14, 2020 3:28 PM
I asked a similar question on stackoverflow. A summary of a response has been included below (as well as a link to the question). The poster indicates that Slurm makes no effort to transfer files and speculates that pcluster has been configured to allow for file exchange. Is this the case? If so, what is that mechanism? Is it encrypted?
Link to stackoverflow: https://stackoverflow.com/questions/65225099/what-mechanism-does-slurm-use-to-sync-files-between-compute-nodes-and-the-master. Copy of answer provided below:
Slurm will assume that there is a common, shared, filesystem available on all compute nodes and will take that as a prerequisite. Often, clusters will have a "home" filesystem, using technologies such as NFS, GPFS, Lustre, GlusterFS, BeeGFS, AndrewFS, etc, along with other filesystems with different performances/reliability tradeoffs.
But Slurm will not make any effort to transfer files to/from compute nodes, except for the submission script.
In your case, this is most probably setup by the procedure you used to spin up the virtual cluster. Indeed, in the blog post you refer to, the configuration file has a line fsx_settings = parallel-fs that seems to indicate there is a parallel filesystem setup. It ifs further configured in the [fsx parallel-fs] section. From reading the AWS documentation, it could be a Lustre filesystem.
As for encryption, it probably isn't as this type of filesystem is designed for performance on private networks, not for security on WANs. The Amazon procedure most probably configures a private network for the compute nodes.
Edited by: ProlucidDavid on Dec 10, 2020 6:32 AM
Hi AWS-Rex,
Thank you for your clarification, I've confirmed that certain directories are being shared by looking at /etc/exports. I have also confirmed that an NFS service is running. This would explain why files are synchronized even though slurm takes no responsibility for that.
Going forward, I need to ensure that all communication between the head and compute node be encrypted. With that in mind do you have any insight on the following questions?
- It looks like there are several ways to add security to NFS, but I can't find any discussion on the NFS setup in pcluster. Do you know if it is setup with some form of security? Is there a link to documentation on this configuration?
- As you've mentioned pcluster supports EFS and FSx. If I were to add a config section for one of those files systems in the global pcluster config, would they be used instead of NFS? Or would they be used in parallel, with NFS managing the user accounts and the other filesystems used for the shared_dir specified in the pcluster config? https://docs.aws.amazon.com/parallelcluster/latest/ug/global.html
- Do you have any additional insight on what the encrypted flag of efs section of pcluster would accomplish? In particular would it lead to encryption in transit? https://docs.aws.amazon.com/parallelcluster/latest/ug/efs-section.html
Thanks for your help!
ProlucidDavid,
Regarding the security with which we configure NFS: in general system defaults are used, and no additional configuration for the sake of added security is performed. This isn't currently documented. That said, I'll create this as a feature request.
Regarding the use of EFS or FSx for more security: the use of these file systems would not prevent NFS from being used to manage the home and slurm configuration file directories. They would be used in parallel with NFS.
Regarding encryption of EFS data in transit: according to https://docs.aws.amazon.com/efs/latest/ug/encryption.html#encryption-in-transit , "You can enable encryption of data at rest when creating an Amazon EFS file system. You can enable encryption of data in transit when you mount the file system." When ParallelCluster creates the EFS file system and the encrypted
configuration directive is set to true
, data is encrypted at rest. Unfortunately I don't think we currently support encryption of data in transit.
~Tim
Relevant content
- Accepted Answerasked 10 months ago
- Accepted Answerasked 3 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 months ago