Skip to content

Custom EFS with CloudFormation and Sagemaker

0

I'm trying to attach a custom image to a Sagemaker Domain for use in multiple spaces, the whole thing done as IaC. I'm unable to replicate what I've done manually with the AWS CLI in Sagemaker.

My current setup is this - any pointers / help would be nice.

Currently I see "Internal Error" when trying to launch the resulting Sagemaker Space with custom EFS attached - it launches okay without any custom EFS attached. There are no CloudWatch logs for the failure. (I've seen this Q&A - I'm still no closer to a resolution though: https://repost.aws/questions/QUntgIa1UMQ_i6AsYmb1Y3dw/sagemaker-studio-with-custom-efs-permission-denied-for-write-operations)

I have EFS mount targets for all subnets of my VPC (security groups are added during CloudFormation stack creation) The same EFS volume has a sub-directory owned by UID: 200001 and GID: 1001. EC2 instance session below.

Last login: Sun Nov 10 13:32:59 2024 from <my-ip>
[ec2-user@ip-... ~]$ sudo mount -t efs -o tls <my-efs-id>:/ /mnt/efs/image_share
[ec2-user@ip-...~]$ cd /mnt/efs/image_share
[ec2-user@ip-... image_share]$ ls -al
total 8
drwxr-xr-x. 3 root   root 6144 Oct  2 20:46 .
drwxr-xr-x. 4 root   root   36 Oct  2 20:44 ..
drwxr-xr-x. 2 200001 1001 6144 Oct  2 20:46 space-user
[ec2-user@ip-... image_share]$ 

My CloudFormation template looks like this (a mix of IaC generator and my own hacking / testing, manual obfuscation...):

Parameters:
  DomainName:
    Type: String
    Default: ai-proto-domain
  DomainUserName:
    Type: String
    Default: space-user
  DomainSpaceName:
    Type: String
    Default: ai-proto-jupyter-space
  VpcId:
    Type: AWS::EC2::VPC::Id
    Default: my-vpc
  SubnetIds:
    Type: CommaDelimitedList
    Default: my-subnet1, my-subnet2, my-subnet3
  DefaultSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Default: default
  AuxiliaryEFS:
    Type: String
    Default: my-shared-efs

Resources:
  SMExecutionRole:
    Type: AWS::IAM::Role
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      Path: /service-role/
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSageMakerCanvasFullAccess
        - arn:aws:iam::aws:policy/AmazonSageMakerCanvasDataPrepFullAccess
        - arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
        - arn:aws:iam::aws:policy/AmazonSageMakerCanvasAIServicesAccess
        - arn:aws:iam::aws:policy/AmazonElasticFileSystemClientReadWriteAccess
        - arn:aws:iam::aws:policy/service-role/AWSLambdaRole
        - !Ref SMS3Policy
        - !Ref EFSAdminPolicy
        - !Ref SMGlueMinimalPolicy
      MaxSessionDuration: 3600
      RoleName: !Sub ${DomainName}-ExecutionRole
      Description: SageMaker execution role created by CloudFormation
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service:
                - sagemaker.amazonaws.com
                - lambda.amazonaws.com

  SMGlueMinimalPolicy:
    Type: AWS::IAM::ManagedPolicy
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      ManagedPolicyName: !Sub ${DomainName}-GlueMinimalPolicy
      Path: /service-role/
      Description: Minimal Glue access policy for SageMaker SQL editor
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - glue:GetConnections
            Resource: '*'

  SMS3Policy:
    Type: AWS::IAM::ManagedPolicy
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      ManagedPolicyName: !Sub ${DomainName}-ExecutionPolicy
      Path: /service-role/
      Description: S3 access policy created by CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Resource:
              - arn:aws:s3:::*
            Action:
              - s3:GetObject
              - s3:PutObject
              - s3:DeleteObject
              - s3:ListBucket
            Effect: Allow

  EFSAdminPolicy:
    Type: AWS::IAM::ManagedPolicy
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      ManagedPolicyName: !Sub ${DomainName}-EFSAdminPolicy
      Path: /service-role/
      Description: EFS administrative access policy created by CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - elasticfilesystem:DescribeMountTargetSecurityGroups
              - elasticfilesystem:ModifyMountTargetSecurityGroups
            Resource: '*'

  sgDomain2EFS:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for SageMaker Domain to EFS access
      VpcId: !Ref VpcId

  sgDomain2EFSEgress:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      IpProtocol: tcp
      FromPort: 2049
      ToPort: 2049
      DestinationSecurityGroupId: !Ref sgEFS2Domain
      GroupId: !Ref sgDomain2EFS

  sgEFS2Domain:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for EFS to SageMaker Domain access
      VpcId: !Ref VpcId

  sgEFS2DomainIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      IpProtocol: tcp
      FromPort: 2049
      ToPort: 2049
      SourceSecurityGroupId: !Ref sgDomain2EFS
      GroupId: !Ref sgEFS2Domain

  UpdateEFSMountTargets:
      Type: Custom::UpdateEFSMountTargets
      DependsOn:
        - sgEFS2Domain
        - UpdateEFSMountTargetsFunction
        - sgEFS2DomainIngress
      Properties:
        ServiceToken: !GetAtt UpdateEFSMountTargetsFunction.Arn
        FileSystemId: !Ref AuxiliaryEFS
        SecurityGroupId: !Ref sgEFS2Domain
        Version: '1.0'

  UpdateEFSMountTargetsFunction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.9
      Handler: index.handler
      Role: !GetAtt SMExecutionRole.Arn
      Timeout: 300
      Code:
        ZipFile: |
          import boto3
          import cfnresponse

          def handler(event, context):
              try:
                  efs = boto3.client('efs')
                  props = event['ResourceProperties']

                  for mt in efs.describe_mount_targets(FileSystemId=props['FileSystemId'])['MountTargets']:
                      groups = efs.describe_mount_target_security_groups(MountTargetId=mt['MountTargetId'])['SecurityGroups']
                      if event['RequestType'] in ('Create', 'Update') and props['SecurityGroupId'] not in groups:
                          groups.append(props['SecurityGroupId'])
                      elif event['RequestType'] == 'Delete' and props['SecurityGroupId'] in groups:
                          groups.remove(props['SecurityGroupId'])
                      efs.modify_mount_target_security_groups(MountTargetId=mt['MountTargetId'], SecurityGroups=groups)

                  cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
              except Exception as e:
                  cfnresponse.send(event, context, cfnresponse.FAILED, {'error': str(e)})

  SMDomain:
    Type: AWS::SageMaker::Domain
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      VpcId: !Ref VpcId
      DomainName: !Ref DomainName
      DefaultUserSettings:
        ExecutionRole: !GetAtt SMExecutionRole.Arn
        CustomFileSystemConfigs:
          - EFSFileSystemConfig:
              FileSystemId: !Ref AuxiliaryEFS
              FileSystemPath: /space-user
        CustomPosixUserConfig:
          Uid: 200001
          Gid: 1001
      DefaultSpaceSettings:
        ExecutionRole: !GetAtt SMExecutionRole.Arn
      SubnetIds: !Ref SubnetIds
      AuthMode: IAM
      DomainSettings:
        SecurityGroupIds:
          - !Ref DefaultSecurityGroupId
          - !Ref sgDomain2EFS

  SMUserProfile:
    Type: AWS::SageMaker::UserProfile
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    DependsOn: SMDomain
    Properties:
      DomainId: !Ref SMDomain
      UserProfileName: !Ref DomainUserName

  SMJupyterSpace:
    Type: AWS::SageMaker::Space
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    DependsOn:
      - SMUserProfile
      - UpdateEFSMountTargets
    Properties:
      DomainId: !Ref SMDomain
      SpaceName: !Ref DomainSpaceName
      SpaceSharingSettings:
        SharingType: Private
      OwnershipSettings:
        OwnerUserProfileName: !Ref DomainUserName
      SpaceSettings:
        AppType: JupyterLab
        JupyterLabAppSettings:
          DefaultResourceSpec:
            InstanceType: ml.t3.medium
            SageMakerImageArn: arn:aws:sagemaker:eu-west-2:021081402939:image/sagemaker-distribution-cpu-v1
        CustomFileSystems:
          - EFSFileSystem:
              FileSystemId: !Ref AuxiliaryEFS
1 Answer
0

When starting a SageMaker Space with a custom EFS, the "Internal Error" is frequently caused by a permissions problem at the EFS level rather than just the IAM level. Although there are many moving components in your setup, the most frequent cause of this problem is that the security group for the SageMaker domain is not given ingress access to the EFS mount targets.

A two-way security group relationship is established between sgDomain2EFS and sgEFS2Domain by your CloudFormation template. But the only thing your UpdateEFSMountTargets custom Lambda function is doing is adding sgEFS2Domain to the EFS mount targets. This implies: The EFS mount targets on port 2049 can be reached by your SageMaker Domain (using sgDomain2EFS). Only traffic from sgDomain2EFS is allowed to reach the EFS mount targets (using sgEFS2Domain). The crucial component that is missing is the ability of the internal infrastructure of the SageMaker service, which provides the Space, to connect to your EFS on your behalf. This means that in addition to the security group for your domain, the EFS mount targets must permit access from the SageMaker managed security groups. Changing your EFS Mount Target Security Groups is the suggested remedy. Making sure your EFS mount targets have a security group that permits ingress from the whole CIDR range of your VPC's subnets where the Domain resides is the simplest and most reliable solution. This will enable communication between the SageMaker internal services and your domain.

Step 1: Replace sgEFS2Domain with a new security group for EFS: EFSSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for EFS allowing access from the entire VPC VpcId: !Ref VpcId SecurityGroupIngress: - IpProtocol: tcp FromPort: 2049 ToPort: 2049 CidrIp: !GetAtt VPC.CidrBlock # This allows all resources in the VPC to access EFS

If you must restrict it further, use the subnet CIDRs instead of the whole VPC.

SecurityGroupIngress:

- IpProtocol: tcp

FromPort: 2049

ToPort: 2049

CidrIp: !Ref 'Subnet1Cidr'

- IpProtocol: tcp

FromPort: 2049

ToPort: 2049

CidrIp: !Ref 'Subnet2Cidr'

- IpProtocol: tcp

FromPort: 2049

ToPort: 2049

CidrIp: !Ref 'Subnet3Cidr'

Step 2: Update your Custom Lambda Resource (UpdateEFSMountTargets): Make sure the new Lambda function and security group are referenced by the ServiceToken and Security GroupId. The logic used by your Lambda function to change the mount targets is sound; it simply needs to apply this new, more lenient security group.

Alternative (More Secure but Complex) Solution: You must locate and permit the particular SageMaker service security groups if granting access to the entire VPC is too permissive. These are frequently difficult to expose in CloudFormation and are managed by AWS. Because it can be brittle and change, this path is not as advised.

Extra Checks: Following the above primary fix, confirm these typical pitfalls as well: IAM Trust Policy: Sagemaker.amazonaws.com is listed as a trusted entity by your SMExecutionRole, which is accurate.

EFS Permissions: The UID/GID (200001/1001) must be able to read and write to the /space-user directory's POSIX permissions. It's ideal that your ls -al output indicates that user owns the directory.

Resource Dependencies: Verify again that your SMJupyterSpace resource has a DependsOn attribute that is explicitly waiting for the successful completion of the UpdateEFSMountTargets custom resource. This is already included in your template, which is good.

In conclusion, begin by resetting your EFS security group to permit NFS traffic (TCP 2049) from the SageMaker Domain's subnets or the CIDR block of your VPC. By enabling the SageMaker provisioning service to mount your EFS filesystem during Space creation, this will most likely fix the "Internal Error."

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.