bootstrap failure due to requirement of arm64 version of numpy required for r6 instances?lg...
Was trying to upgrade to the latest r6 instances from r5s and ran into an issue with installing numpy in our bootstrap script via pip.
Found[ this post](https://repost.aws/questions/QUdF4dL0k9RTeAZaUFiPDJCw/emr-bootstrap-script-with-pip-numpy-installation-fails-on-r-6-instances) that is related to my issue. Was anyone able to resolve this without building your own wheel file of the arm64 version of numpy?
EC2/EMR Cluster Config
```
Release label: emr-6.5.0
Instance Type: r6gd.8xlarge
```
Snippet of the bootstrap
```
#!/bin/bash
# python version
pyv="$(python3 -V 2>&1)"
echo "Python version: $pyv"
# misc code to link up the requirements.txt
echo "`date -u` install python dependencies"
#Install Python deps
sudo python3 -m pip install wheel
sudo python3 -m pip install -r requirements.txt
```
requirements.txt
```
boto3==1.18.46
Cython==0.29.24
pandas==1.3.3
numpy==1.21.2
```
Log Output
```
+ echo 'Python version: Python 3.7.10'
...
+ echo 'Thu Sep 29 22:16:46 UTC 2022 install python dependencies'
+ sudo python3 -m pip install wheel
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
WARNING: The script wheel is installed in '/usr/local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
+ sudo python3 -m pip install -r job-requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
ERROR: Command errored out with exit status 1:
command: /bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
cwd: None
Complete output (866 lines):
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Ignoring numpy: markers 'python_version < "3.7"' don't match your environment
Collecting cython>=0.29
Using cached Cython-0.29.32-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (1.8 MB)
Collecting numpy==1.16.0
Downloading numpy-1.16.0.zip (5.1 MB)
Collecting setuptools
Downloading setuptools-65.4.0-py3-none-any.whl (1.2 MB)
Collecting setuptools_scm
Downloading setuptools_scm-7.0.5-py3-none-any.whl (42 kB)
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting packaging>=20.0
Downloading packaging-21.3-py3-none-any.whl (40 kB)
Collecting tomli>=1.0.0
Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting typing-extensions
Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Collecting importlib-metadata; python_version < "3.8"
Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting zipp>=0.5
Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)
...
_configtest.c:1:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
compilation terminated.
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py", line 415, in <module>
setup_package()
File "/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py", line 407, in setup_package
setup(**metadata)
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/core.py", line 171, in setup
return old_setup(**new_attr)
File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 165, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/install.py", line 62, in run
r = self.setuptools_run()
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run
return distutils_install.run(self)
File "/usr/lib64/python3.7/distutils/command/install.py", line 556, in run
self.run_command('build')
File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build.py", line 47, in run
old_build.run(self)
File "/usr/lib64/python3.7/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 148, in run
self.build_sources()
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 165, in build_sources
self.build_extension_sources(ext)
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 322, in build_extension_sources
sources = self.generate_sources(sources, ext)
File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 375, in generate_sources
source = func(extension, build_dir)
File "numpy/core/setup.py", line 423, in generate_config_h
moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
File "numpy/core/setup.py", line 47, in check_types
out = check_types(*a, **kw)
File "numpy/core/setup.py", line 281, in check_types
"install {0}-dev|{0}-devel.".format(python))
SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
----------------------------------------
ERROR: Command errored out with exit status 1: /bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py'"'"'; __file__='"'"'/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /mnt/tmp/pip-record-paofd9vx/install-record.txt --single-version-externally-managed --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --compile --install-headers /mnt/tmp/pip-build-env-yy928eo_/overlay/include/python3.7m/numpy Check the logs for full command output.
----------------------------------------
ERROR: Command errored out with exit status 1: /bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output.
```lg...
