- Newest
- Most votes
- Most comments
When starting a SageMaker Space with a custom EFS, the "Internal Error" is frequently caused by a permissions problem at the EFS level rather than just the IAM level. Although there are many moving components in your setup, the most frequent cause of this problem is that the security group for the SageMaker domain is not given ingress access to the EFS mount targets.
A two-way security group relationship is established between sgDomain2EFS and sgEFS2Domain by your CloudFormation template. But the only thing your UpdateEFSMountTargets custom Lambda function is doing is adding sgEFS2Domain to the EFS mount targets. This implies: The EFS mount targets on port 2049 can be reached by your SageMaker Domain (using sgDomain2EFS). Only traffic from sgDomain2EFS is allowed to reach the EFS mount targets (using sgEFS2Domain). The crucial component that is missing is the ability of the internal infrastructure of the SageMaker service, which provides the Space, to connect to your EFS on your behalf. This means that in addition to the security group for your domain, the EFS mount targets must permit access from the SageMaker managed security groups. Changing your EFS Mount Target Security Groups is the suggested remedy. Making sure your EFS mount targets have a security group that permits ingress from the whole CIDR range of your VPC's subnets where the Domain resides is the simplest and most reliable solution. This will enable communication between the SageMaker internal services and your domain.
Step 1: Replace sgEFS2Domain with a new security group for EFS: EFSSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security group for EFS allowing access from the entire VPC VpcId: !Ref VpcId SecurityGroupIngress: - IpProtocol: tcp FromPort: 2049 ToPort: 2049 CidrIp: !GetAtt VPC.CidrBlock # This allows all resources in the VPC to access EFS
If you must restrict it further, use the subnet CIDRs instead of the whole VPC.
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 2049
ToPort: 2049
CidrIp: !Ref 'Subnet1Cidr'
- IpProtocol: tcp
FromPort: 2049
ToPort: 2049
CidrIp: !Ref 'Subnet2Cidr'
- IpProtocol: tcp
FromPort: 2049
ToPort: 2049
CidrIp: !Ref 'Subnet3Cidr'
Step 2: Update your Custom Lambda Resource (UpdateEFSMountTargets): Make sure the new Lambda function and security group are referenced by the ServiceToken and Security GroupId. The logic used by your Lambda function to change the mount targets is sound; it simply needs to apply this new, more lenient security group.
Alternative (More Secure but Complex) Solution: You must locate and permit the particular SageMaker service security groups if granting access to the entire VPC is too permissive. These are frequently difficult to expose in CloudFormation and are managed by AWS. Because it can be brittle and change, this path is not as advised.
Extra Checks: Following the above primary fix, confirm these typical pitfalls as well: IAM Trust Policy: Sagemaker.amazonaws.com is listed as a trusted entity by your SMExecutionRole, which is accurate.
EFS Permissions: The UID/GID (200001/1001) must be able to read and write to the /space-user directory's POSIX permissions. It's ideal that your ls -al output indicates that user owns the directory.
Resource Dependencies: Verify again that your SMJupyterSpace resource has a DependsOn attribute that is explicitly waiting for the successful completion of the UpdateEFSMountTargets custom resource. This is already included in your template, which is good.
In conclusion, begin by resetting your EFS security group to permit NFS traffic (TCP 2049) from the SageMaker Domain's subnets or the CIDR block of your VPC. By enabling the SageMaker provisioning service to mount your EFS filesystem during Space creation, this will most likely fix the "Internal Error."
answered a year ago
