How to bring customer S3 bucket to Pcluster3.8.0 FSX/Lustre

0

I have a pre-existing S3 bucket-in-different-account (which we use daily for old 2.x PCluster fsx-lustre). We'd like to use that pre-existing S3 bucket and use it for new clusters using fsx-lustre in PCluster 3.8.0.

WHAT-DOESN"T-WORK So far I've resorted to hand-jamming permissions on random pcluster3 roles in order to give the PCluster3 ui access to the customers S3 bucket. All that gets me is PCUI sends the CloudFormation w/params for cluster creation. But, that fails in CloudFormation with...

Amazon FSx is unable to create Service-Linked-Role to access the S3 bucket. 
Ensure the IAM role or user you are using has the required permissions. 
For more details, visit https://docs.aws.amazon.com/fsx/latest/LustreGuide/setting-up.html#fsx-adding-permissions-s3. 
(Service: AmazonFSx; Status Code: 400; Error Code: BadRequest; Request ID: eb6ce2e0-6c1f-4c6f-bb7c-d71cad7ea27c; Proxy: null)

and the Stack is reverted...

WHAT DOES WORK If I abandon the customers S3 bucket (again in a different account), and I use the s3 bucket that pcluster3 creates as part of the UI thing parallelcluster-REDACTED-v1-do-not-delete I'm able to get a successfully created cluster with an fsx/ mount.
HOWEVER, I'd like to avoid copying everything over from the customer's S3 bucket to this S3 bucket.

alfreds
질문됨 3달 전112회 조회
3개 답변
1
수락된 답변

I made some progress, at least I got past that error message and onto the next error.

I already had cross-acount permissions working as I stated in the comments to both initial answers so that wasn't my issue.

I spelled out in detail what I had to do in my bug https://github.com/aws/aws-parallelcluster/issues/6122

basically, you have to go heavy spelunking into finding which pcluster3 role is needing the service-linked-role and add a statement to the already full (10 max) policy role.

Unfortunately, you'll then get the next productivity killer

Amazon FSx is unable to import objects from the linked data repository. Please file a ticket at https://console.aws.amazon.com/support/home#/. While filing your ticket, please include your file system ID and name of the linked data repository.

So now I'm onto that.

alfreds
답변함 2달 전
profile picture
전문가
검토됨 2달 전
1

To integrate your pre-existing S3 bucket from a different account with PCluster 3.8.0 FSx/Lustre, you'll need to establish cross-account access and ensure the appropriate permissions are set up. Follow these steps:

  1. Set Up Cross-Account Access:

    • Create a role in your account (let's call it Account A).
    • Create another role in the target account (Account B) with a trusted policy that allows access from Account A.
    • This will enable seamless communication between the two accounts.
  2. Grant Access between Accounts:

    • Define a trusted policy in Account A to allow access from Account B.
    • Similarly, create a trusted policy in Account B to allow access from Account A.

    Trusted Policy for Account A:

    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID-A:role/[ROLE_NAME_ACCOUNT_A]"
      }
    }

    Trusted Policy for Account B:

    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::PRODUCTION-ACCOUNT-ID-B:role/[ROLE_NAME_ACCOUNT_A]"
      }
    }
  3. Configure Permissions for Amazon FSx:

    • Follow the guidelines provided in the Amazon FSx for Lustre documentation to set up the necessary permissions for accessing S3 buckets. You can find detailed instructions here.

Additional resources:

profile picture
전문가
답변함 3달 전
  • I think I failed to stress the part about "[...] which we use daily for old 2.x PCluster fsx-lustre) [...]" What I meant there was... We already have cross account FSX/Lustre working like a charm for PCluster 2. we use it daily (as in today and have for years). So, really all I'm trying to do is succeed with PCluster 3 in a way that PCluster 2 already allowed for.

    I think what might be missing on the PCluster 3 side is the ability/knowlege about where to add the same policy that I added for our pcluster2 role, which looks something like this (only not redacted ) ...

    {
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:PutObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws-us-gov:s3:::REDACTED",
                    "arn:aws-us-gov:s3:::REDACTED/*"
                ],
                "Sid": "S3REDACTED"
            },
    
0

Hello, it seems that you would like to set up fsx cross account S3 bucket access. That will required some additional settings for cross-account bucket access. Could you check if the following link helpful? https://repost.aws/questions/QUPA2hF7XSRamoN8sXgyovBw/how-to-access-bucket-from-another-account-to-fsx

답변함 3달 전
  • COPY/PASTING my response to similar answer...

    I think I failed to stress the part about "[...] which we use daily for old 2.x PCluster fsx-lustre) [...]" What I meant there was... We already have cross account FSX/Lustre working like a charm for PCluster 2. we use it daily (as in today and have for years). So, really all I'm trying to do is succeed with PCluster 3 in a way that PCluster 2 already allowed for.

    I think what might be missing on the PCluster 3 side is the ability/knowlege about where to add the same policy that I added for our pcluster2 role, which looks something like this (only not redacted ) ...

    {
                "Action": [
                    "s3:Get*",
                    "s3:List*",
                    "s3:PutObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws-us-gov:s3:::REDACTED",
                    "arn:aws-us-gov:s3:::REDACTED/*"
                ],
                "Sid": "S3REDACTED"
            },
    

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠