Does anyone have OpenZFS filesystem with AWS Parallelcluster working ?

0

Q: Has anyone created an OpenZFS volume in FSx and been able to mount the volume using the pcluster3 cli in AWS Parallelcluster ? It doesn't seem to work. I created an OpenZFS filesystem, but I can't get it to mount using from in the pcluster cli. I do not have this issue with other filesystems. The YAML descriptors for the filesystem mount looks like this:

SharedStorage:
  - Name: modelingtest
    StorageType: FsxLustre
    MountDir: /fsx
    FsxLustreSettings:
      FileSystemId: fs-045ffe08c17984010

It seems the CLI spits out a non-descriptive error before the cluster build even starts. This looks like a cli bug to me.

pcluster create-cluster --cluster-configuration ./bluefishtestfsx.yaml --cluster-name testami --region us-east-1
{
  "message": "'NoneType' object has no attribute 'get'"
}

The documenation for this is interesting too. It says you need to specifiy the storage size unless the "FileSystemID" is specified. https://docs.aws.amazon.com/parallelcluster/latest/ug/SharedStorage-v3.html

If I add the filesystem size, we get a different error which makes sense.

 pcluster create-cluster --cluster-configuration ./bluefishtestfsx.yaml --cluster-name testami --region us-east-1
{
  "configurationValidationErrors": [
    {
      "level": "ERROR",
      "type": "ConfigSchemaValidator",
      "message": "[('SharedStorage', {0: {'FsxLustreSettings': {'_schema': ['storage_capacity is ignored when an existing Lustre file system is specified.']}}})]"
    }
  ],
  "message": "Invalid cluster configuration."
}
zeekus
gefragt vor 2 Jahren307 Aufrufe
2 Antworten
0
Akzeptierte Antwort

Today, ParallelCluster only supports mounting FSx for OpenZFS file systems using custom bootstrap actions.

You need to include your mount -t nfs -o nfsvers=4.1 fs-12345.fsx.us-east-1.amazonaws.com:/fsx/ /localpath command in a script, and reference that script under CustomActions: OnNodeConfigured on HeadNode and Scheduling:SlurmQueues in your ParallelCluster configuration file.

AWS
delwio
beantwortet vor 2 Jahren
0

AWS ParallelCluster launched support for FSx OpenZFS in 3.2.0. You can read more about it here: https://aws.amazon.com/blogs/hpc/expanded-filesystems-support-in-aws-parallelcluster-3-2/

Essentially you pre-create the filesystem and mount it like:

- MountDir: /openzfs
    Name: OpenZFS
    StorageType: FsxOpenZfs
    FsxOpenZfsSettings:
      VolumeId: fsvol-02222222222222222
swsmith
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen