Sagemaker Training Job. Python modules installation Error

Question

I have a problem with Python module installation that requires pre-installation of another module.  Both modules were added to the `requirement.txt` file. However, the error occurs when installing main module:
```message
2022-07-29 01:18:26.460132: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.
"2022-07-29 01:18:26.470589: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped."
2022-07-29 01:18:26.765280: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.
"2022-07-29 01:18:31,908 sagemaker-training-toolkit INFO     Imported framework sagemaker_tensorflow_container.training"
"2022-07-29 01:18:31,917 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)"
"2022-07-29 01:18:33,117 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:"
/usr/local/bin/python3.9 -m pip install -r requirements.txt
Collecting Cython==0.29.31
Downloading Cython-0.29.31-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 33.1 MB/s eta 0:00:00
Requirement already satisfied: wheel==0.37.1 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (0.37.1)
Collecting scikit-image==0.19.2
Downloading scikit_image-0.19.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 83.1 MB/s eta 0:00:00
Collecting parallelbar==0.1.19
Downloading parallelbar-0.1.19-py3-none-any.whl (5.6 kB)
Collecting albumentations==1.0.3
Downloading albumentations-1.0.3-py3-none-any.whl (98 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.7/98.7 kB 6.6 MB/s eta 0:00:00
Collecting tensorflow_addons==0.16.1
Downloading tensorflow_addons-0.16.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 54.4 MB/s eta 0:00:00
Requirement already satisfied: tensorflow-io==0.24.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 7)) (0.24.0)
Requirement already satisfied: tensorboard==2.8.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 8)) (2.8.0)
Collecting universal-pathlib==0.0.12
Downloading universal_pathlib-0.0.12-py3-none-any.whl (19 kB)
Collecting setuptools==63.2.0
Downloading setuptools-63.2.0-py3-none-any.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 58.9 MB/s eta 0:00:00
Collecting pynanosvg==0.3.1
Downloading pynanosvg-0.3.1.tar.gz (346 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 346.0/346.0 kB 17.5 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
"error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File """", line 2, in 
        File """", line 34, in 
        File ""/tmp/pip-install-1mt2gkfy/pynanosvg_d6162ffce95948abb4262061a011908c/setup.py"", line 2, in 
          from Cython.Build import cythonize
      ModuleNotFoundError: No module named 'Cython'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip."
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
"note: This is an issue with the package mentioned above, not pip."
hint: See above for details.
[notice] A new release of pip available: 22.1.2 -> 22.2.1
"[notice] To update, run: pip install --upgrade pip"
"2022-07-29 01:18:36,187 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code."
"2022-07-29 01:18:36,187 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 1 from exiting process."
"2022-07-29 01:18:36,188 sagemaker-training-toolkit ERROR    Reporting training FAILURE"
"2022-07-29 01:18:36,188 sagemaker-training-toolkit ERROR    InstallRequirementsError:"
ExitCode 1
"ErrorMessage ""      ModuleNotFoundError: No module named 'Cython'
       [end of output]      note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed  × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details."""
"Command ""/usr/local/bin/python3.9 -m pip install -r requirements.txt"""
"2022-07-29 01:18:36,188 sagemaker-training-toolkit ERROR    Encountered exit_code 1"

Answer

Since Cython does seem to be downloaded before the error, I suspect the problem is something in other packages' install process requiring it before pip is done installing it. It looks like others have found similar (non-SageMaker-specific) issues with Cython e.g. [here](https://stackoverflow.com/questions/61750900/cython-modulenotfounderror-even-though-it-was-just-installed-via-requirements) and [here](https://github.com/h5py/h5py/issues/535).

Things I would suggest to try:

1. **Explicitly specify** Cython (maybe without a version at first) right at the top of your requirements.txt file if you're not already - just in case this can convince pip to treat it properly.

2. **Customize** the TensorFlow container image you're targeting to pre-install Cython.

If you're not sure what base container URI you're using, you can fetch it with `sagemaker.image_uris.retrieve(...)` ([doc here](https://sagemaker.readthedocs.io/en/stable/api/utility/image_uris.html#sagemaker.image_uris.retrieve)).

From that, you can create a minimal Dockerfile something like

```Dockerfile
FROM XYZ.dkr.ecr.ABC.amazonaws.com/...
RUN pip install Cython==0.29.31
```

Once you build this customized container image, and push it to Amazon ECR in your AWS account & region, you can use it by setting the `image_uri` [parameter](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#sagemaker.tensorflow.estimator.TensorFlow) in your TensorFlow Estimator. Note that the frameworks typically have separate container images for training vs serving, and GPU vs CPU-only, so you may need to create a pair of containers if wanting to do inference too.

If you're working inside SageMaker Studio you won't directly be able to `docker build`, but you can install the [sm-docker build solution](https://aws.amazon.com/blogs/machine-learning/using-the-amazon-sagemaker-studio-image-build-cli-to-build-container-images-from-your-studio-notebooks/) based on AWS CodeBuild. The *"Prepare custom training and inference containers"* section of [this notebook](https://github.com/aws-samples/amazon-textract-transformer-pipeline/blob/5f9ecb8a1d861bae850403b4d9937d6f9ea6a9c2/notebooks/2.%20Model%20Training.ipynb) gives an example of similar approach.

3. If you'd really like to avoid touching containers and ECR, you could instead remove your requirements.txt and install dependencies **within the script** via something like `subprocess.check_call(["pip", "install", ...])`. It's hacky, but this way you could run a `pip install` just for Cython first... **Then** install all the other dependencies in one other command.

Sagemaker Training Job. Python modules installation Error

Relevant content