Sagemaker: OSError: /lib64/libm.so.6: version `GLIBC_2.27' not found

I have paid support, but decided to open post here too. Sometimes paid support doesn't know paid support for various reasons. Basically, I have want to use pyg_lib and import in my notebook. Here is my general configurations for my sagemaker notebook instance:

• ml.t3.medium
• Amazon Linux 2, Jupyter Lab 3

When I start the notebook, I used the AWS provided conda_pytorch_p310 kernel as well as the AWS provided condo_python3 kernel. They both resulted in the below. I also attempted to create a custom kernel using conda environment and specified python 3.12, it also resulted in below.

==============================================

I've installed initial dependencies successfully:

!pip install torch torch-geometric

Now, I try to install desired packages and also succeed:

%%bash
torch_version=$(python -c "import torch; print(torch.__version__)")
echo "$torch_version"

pip install pyg-lib -f "https://data.pyg.org/whl/torch-${torch_version}+cpu.html"

2.2.0
Looking in links: https://data.pyg.org/whl/torch-2.2.0+cpu.html
Collecting pyg-lib
  Downloading https://data.pyg.org/whl/torch-2.2.0%2Bcpu/pyg_lib-0.4.0%2Bpt22cpu-cp310-cp310-linux_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 18.4 MB/s eta 0:00:00:00:01
Installing collected packages: pyg-lib
Successfully installed pyg-lib-0.4.0+pt22cpu

However, when I try to import the package:

import pyg_lib

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[9], line 1
----> 1 import pyg_lib

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/pyg_lib/__init__.py:39
     35     else:
     36         torch.ops.load_library(spec.origin)
---> 39 load_library('libpyg')
     42 def cuda_version() -> int:
     43     r"""Returns the CUDA version for which :obj:`pyg_lib` was compiled with.
     44 
     45     Returns:
     46         (int): The CUDA version.
     47     """

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/pyg_lib/__init__.py:36, in load_library(lib_name)
     34     warnings.warn(f"Could not find shared library '{lib_name}'")
     35 else:
---> 36     torch.ops.load_library(spec.origin)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/_ops.py:933, in _Ops.load_library(self, path)
    928 path = _utils_internal.resolve_library_path(path)
    929 with dl_open_guard():
    930     # Import the shared library into the process, thus running its
    931     # static (global) initialization code in order to register custom
    932     # operators with the JIT.
--> 933     ctypes.CDLL(path)
    934 self.loaded_libraries.add(path)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/ctypes/__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371 self._FuncPtr = _FuncPtr
    373 if handle is None:
--> 374     self._handle = _dlopen(self._name, mode)
    375 else:
    376     self._handle = handle

OSError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/libpyg.so)

I also attempted to run the following commands [1], since in the past installing build-essentials on debian/ubuntu machines fixed glibc related errors. So, running the below commands, then repeating all of the above:

%%bash
yum install make glibc-devel gcc patch

Resulted in the same error:

OSError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/libpyg.so)

I have also attempted to install torch (2.4, 2.3, 2.2, and 2.0), and subsequently pyg-lib [2] at lower versions (both matching each other). However, attempting to import resulted in same error.

---

[1] https://serverfault.com/a/204896
[2] https://github.com/pyg-team/pyg-lib?tab=readme-ov-file#installation