Installing Conda packages in Sagemaker without public internet

Hi,

The tricky parts of the implementation of this blog are the VPC endpoints mentioned in:

Create interface endpoints (via AWS PrivateLink) to access the following 
(and other AWS services you may require):
 - Amazon SageMaker API
 - Amazon SageMaker runtime
 - Amazon S3
 - AWS Security Token Service (AWS STS)
- Amazon CloudWatch

So, did you validate that all your interface endpoints are working correctly? The simplest way to validate them is by adding an EC2 machine to which you will ssh in order to send CLI requests to all the endpoints above.

The timeouts that you mention happen often when such endpoints are not configured properly.

Best,

Didier

Hi,

I run a ML/DS platform team for a large global company and we have similar restrictions on our AWS accounts (no direct internet access e.g. no Internet Gateways, NAT Gateways, Egress-Only Internet Gateways, etc) and all VPC network traffic is centrally controlled through Transit Gateways, firewalls and proxies. As Didier already mentioned, you want to make sure you have VPC endpoints deployed for all the required SageMaker (SM) services, make sure that DNS support and host names are enabled is enabled in your VPC. For any interface endpoints required have a look on at this resource.

The timeout error you are getting indicates a network issue over an access control issue (where you would see some sort of access denied exception returned). Also use the S3 Gateway endpoint over the interface one check that it is routable from SM subnets route table(s) and make sure that the security groups rules for all the interface endpoints allow traffic from the SM subnets or the whole VPC.

A side note, If you are using S3 as a conda registry, you will need to curate and update the S3 bucket with packages yourself, which can be time consuming. To provide a much better Data Science experience you will need to look at some sort of package management solution, which Sagemaker can be allowed to connect to and that will retrieve, cache and preferably scan any external package for malware or vulnerabilities. The blog post you referenced uses AWS CodeArtifact which provides access to python but not conda packages and no scanning capability. Most larger enterprises will invest in a solution such as JFrog Artifactory or Sonatype Nexus which provides centralised configuration and scanning capability.

If you can use python packages (pip) directly instead of conda packages (much larger DS/ML package eco-system),you can look at using AWS CodeArifact and if doing so you will need to additionally add a VPC interface endpoint for this service to your VPC and update your pip.conf to use the AWS CodeArtifact endpoint. This can be done manually or via a SM Lifecycle Configuration. You might also want to look at some sort of package vulnerability scanning tool if going down this path.