Since Cython does seem to be downloaded before the error, I suspect the problem is something in other packages' install process requiring it before pip is done installing it. It looks like others have found similar (non-SageMaker-specific) issues with Cython e.g. here and here.
Things I would suggest to try:
Explicitly specify Cython (maybe without a version at first) right at the top of your requirements.txt file if you're not already - just in case this can convince pip to treat it properly.
Customize the TensorFlow container image you're targeting to pre-install Cython.
If you're not sure what base container URI you're using, you can fetch it with
sagemaker.image_uris.retrieve(...) (doc here).
From that, you can create a minimal Dockerfile something like
FROM XYZ.dkr.ecr.ABC.amazonaws.com/... RUN pip install Cython==0.29.31
Once you build this customized container image, and push it to Amazon ECR in your AWS account & region, you can use it by setting the
image_uri parameter in your TensorFlow Estimator. Note that the frameworks typically have separate container images for training vs serving, and GPU vs CPU-only, so you may need to create a pair of containers if wanting to do inference too.
If you're working inside SageMaker Studio you won't directly be able to
docker build, but you can install the sm-docker build solution based on AWS CodeBuild. The "Prepare custom training and inference containers" section of this notebook gives an example of similar approach.
- If you'd really like to avoid touching containers and ECR, you could instead remove your requirements.txt and install dependencies within the script via something like
subprocess.check_call(["pip", "install", ...]). It's hacky, but this way you could run a
pip installjust for Cython first... Then install all the other dependencies in one other command.
Installation of jupyter notebook on Deep Learning AMI GPU TensorFlow 2.9.1 (Amazon Linux 2)asked 6 days ago
Python cfn_tools module won't load in CodeBuildAccepted Answerasked 8 months ago
GGC v2 component can not find python module awsiotasked a year ago
Do I have to redownload dataset to training job every time I run a Sagemaker Estimator training job?asked 8 months ago
aws lambda - ES6 module error : module is not defined in ES module scopeAccepted Answerasked 7 months ago
Trouble with "Deploy a Web Application on Amazon EC2" training moduleasked 2 months ago
Installing extra modules in PHP 7.3 in Codebuildasked 3 years ago
Sagemaker Training Job. Python modules installation Errorasked 14 days ago
not able to install sshtunnel python module in AWS Glueasked 5 months ago
Sagemaker python sdk installation troublesAccepted Answerasked 2 years ago