How to bring customer S3 bucket to Pcluster3.8.0 FSX/Lustre

0

I have a pre-existing S3 bucket-in-different-account (which we use daily for old 2.x PCluster fsx-lustre). We'd like to use that pre-existing S3 bucket and use it for new clusters using fsx-lustre in PCluster 3.8.0.

WHAT-DOESN"T-WORK So far I've resorted to hand-jamming permissions on random pcluster3 roles in order to give the PCluster3 ui access to the customers S3 bucket. All that gets me is PCUI sends the CloudFormation w/params for cluster creation. But, that fails in CloudFormation with...

Amazon FSx is unable to create Service-Linked-Role to access the S3 bucket. 
Ensure the IAM role or user you are using has the required permissions. 
For more details, visit https://docs.aws.amazon.com/fsx/latest/LustreGuide/setting-up.html#fsx-adding-permissions-s3. 
(Service: AmazonFSx; Status Code: 400; Error Code: BadRequest; Request ID: eb6ce2e0-6c1f-4c6f-bb7c-d71cad7ea27c; Proxy: null)

and the Stack is reverted...

WHAT DOES WORK If I abandon the customers S3 bucket (again in a different account), and I use the s3 bucket that pcluster3 creates as part of the UI thing parallelcluster-REDACTED-v1-do-not-delete I'm able to get a successfully created cluster with an fsx/ mount.
HOWEVER, I'd like to avoid copying everything over from the customer's S3 bucket to this S3 bucket.

alfreds
已提問 3 個月前檢視次數 110 次
3 個答案
1
已接受的答案

I made some progress, at least I got past that error message and onto the next error.

I already had cross-acount permissions working as I stated in the comments to both initial answers so that wasn't my issue.

I spelled out in detail what I had to do in my bug https://github.com/aws/aws-parallelcluster/issues/6122

basically, you have to go heavy spelunking into finding which pcluster3 role is needing the service-linked-role and add a statement to the already full (10 max) policy role.

Unfortunately, you'll then get the next productivity killer

Amazon FSx is unable to import objects from the linked data repository. Please file a ticket at https://console.aws.amazon.com/support/home#/. While filing your ticket, please include your file system ID and name of the linked data repository.

So now I'm onto that.

alfreds
已回答 2 個月前
profile picture
專家
已審閱 2 個月前
1

To integrate your pre-existing S3 bucket from a different account with PCluster 3.8.0 FSx/Lustre, you'll need to establish cross-account access and ensure the appropriate permissions are set up. Follow these steps:

  1. Set Up Cross-Account Access:

    • Create a role in your account (let's call it Account A).
    • Create another role in the target account (Account B) with a trusted policy that allows access from Account A.
    • This will enable seamless communication between the two accounts.
  2. Grant Access between Accounts:

    • Define a trusted policy in Account A to allow access from Account B.
    • Similarly, create a trusted policy in Account B to allow access from Account A.

    Trusted Policy for Account A:

    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID-A:role/[ROLE_NAME_ACCOUNT_A]"
      }
    }

    Trusted Policy for Account B:

    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID-B:role/[ROLE_NAME_ACCOUNT_A]"
      }
    }
  3. Configure Permissions for Amazon FSx:

    • Follow the guidelines provided in the Amazon FSx for Lustre documentation to set up the necessary permissions for accessing S3 buckets. You can find detailed instructions here.

Additional resources:

profile picture
專家
已回答 3 個月前
  • I think I failed to stress the part about "[...] which we use daily for old 2.x PCluster fsx-lustre) [...]" What I meant there was... We already have cross account FSX/Lustre working like a charm for PCluster 2. we use it daily (as in today and have for years). So, really all I'm trying to do is succeed with PCluster 3 in a way that PCluster 2 already allowed for.

    I think what might be missing on the PCluster 3 side is the ability/knowlege about where to add the same policy that I added for our pcluster2 role, which looks something like this (only not redacted ) ...

    {
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:PutObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws-us-gov:s3:::REDACTED",
                    "arn:aws-us-gov:s3:::REDACTED/*"
                ],
                "Sid": "S3REDACTED"
            },
    
0

Hello, it seems that you would like to set up fsx cross account S3 bucket access. That will required some additional settings for cross-account bucket access. Could you check if the following link helpful? https://repost.aws/questions/QUPA2hF7XSRamoN8sXgyovBw/how-to-access-bucket-from-another-account-to-fsx

已回答 3 個月前
  • COPY/PASTING my response to similar answer...

    I think I failed to stress the part about "[...] which we use daily for old 2.x PCluster fsx-lustre) [...]" What I meant there was... We already have cross account FSX/Lustre working like a charm for PCluster 2. we use it daily (as in today and have for years). So, really all I'm trying to do is succeed with PCluster 3 in a way that PCluster 2 already allowed for.

    I think what might be missing on the PCluster 3 side is the ability/knowlege about where to add the same policy that I added for our pcluster2 role, which looks something like this (only not redacted ) ...

    {
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:PutObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws-us-gov:s3:::REDACTED",
                    "arn:aws-us-gov:s3:::REDACTED/*"
                ],
                "Sid": "S3REDACTED"
            },
    

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南