bootstrap failure due to requirement of arm64 version of numpy required for r6 instances?

0

Was trying to upgrade to the latest r6 instances from r5s and ran into an issue with installing numpy in our bootstrap script via pip.

Found this post that is related to my issue. Was anyone able to resolve this without building your own wheel file of the arm64 version of numpy?

EC2/EMR Cluster Config

Release label:   emr-6.5.0
Instance Type:   r6gd.8xlarge

Snippet of the bootstrap

#!/bin/bash

# python version
pyv="$(python3 -V 2>&1)"
echo "Python version: $pyv"

# misc code to link up the requirements.txt

echo "`date -u` install python dependencies"
#Install Python deps
sudo python3 -m pip install wheel
sudo python3 -m pip install -r requirements.txt

requirements.txt

boto3==1.18.46
Cython==0.29.24
pandas==1.3.3
numpy==1.21.2

Log Output

+ echo 'Python version: Python 3.7.10'
 ...
+ echo 'Thu Sep 29 22:16:46 UTC 2022 install python dependencies'
+ sudo python3 -m pip install wheel
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
  WARNING: The script wheel is installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
+ sudo python3 -m pip install -r job-requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
  ERROR: Command errored out with exit status 1:
   command: /bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
       cwd: None
  Complete output (866 lines):
  WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
  Ignoring numpy: markers 'python_version < "3.7"' don't match your environment
  Collecting cython>=0.29
    Using cached Cython-0.29.32-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (1.8 MB)
  Collecting numpy==1.16.0
    Downloading numpy-1.16.0.zip (5.1 MB)
  Collecting setuptools
    Downloading setuptools-65.4.0-py3-none-any.whl (1.2 MB)
  Collecting setuptools_scm
    Downloading setuptools_scm-7.0.5-py3-none-any.whl (42 kB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting packaging>=20.0
    Downloading packaging-21.3-py3-none-any.whl (40 kB)
  Collecting tomli>=1.0.0
    Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
  Collecting typing-extensions
    Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
  Collecting importlib-metadata; python_version < "3.8"
    Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
  Collecting pyparsing!=3.0.5,>=2.0.2
    Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
  Collecting zipp>=0.5
    Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)
  ...
_configtest.c:1:10: fatal error: Python.h: No such file or directory
       #include <Python.h>
                ^~~~~~~~~~
      compilation terminated.
      failure.
      removing: _configtest.c _configtest.o
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py", line 415, in <module>
          setup_package()
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py", line 407, in setup_package
          setup(**metadata)
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/core.py", line 171, in setup
          return old_setup(**new_attr)
        File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 165, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/install.py", line 62, in run
          r = self.setuptools_run()
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run
          return distutils_install.run(self)
        File "/usr/lib64/python3.7/distutils/command/install.py", line 556, in run
          self.run_command('build')
        File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build.py", line 47, in run
          old_build.run(self)
        File "/usr/lib64/python3.7/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 148, in run
          self.build_sources()
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 165, in build_sources
          self.build_extension_sources(ext)
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 322, in build_extension_sources
          sources = self.generate_sources(sources, ext)
        File "/mnt/tmp/pip-install-tl9eju6y/numpy/numpy/distutils/command/build_src.py", line 375, in generate_sources
          source = func(extension, build_dir)
        File "numpy/core/setup.py", line 423, in generate_config_h
          moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
        File "numpy/core/setup.py", line 47, in check_types
          out = check_types(*a, **kw)
        File "numpy/core/setup.py", line 281, in check_types
          "install {0}-dev|{0}-devel.".format(python))
      SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py'"'"'; __file__='"'"'/mnt/tmp/pip-install-tl9eju6y/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /mnt/tmp/pip-record-paofd9vx/install-record.txt --single-version-externally-managed --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --compile --install-headers /mnt/tmp/pip-build-env-yy928eo_/overlay/include/python3.7m/numpy Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /mnt/tmp/pip-build-env-yy928eo_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output.
asked a year ago210 views
1 Answer
0

Hello,

Thank you for reaching out to us. I understand that you are facing issues while installing numpy module from requirements.txt file.

Looking at the error, I can see that the pip is trying to install packages from libraries.h and failed to locate the header file. This can be seen in the error logs as well:


SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.


You can try installing the -devel package for python3.7: # yum instal python3-devel

[Similar Errors] [+] https://stackoverflow.com/questions/21530577/fatal-error-python-h-no-such-file-or-directory [+] https://askubuntu.com/questions/526304/cannot-compile-python-h-perhaps-you-need-to-install-python-devpython-devel

I sincerely hope this helps. If the suggestions above do not help resolve the issue, we might need to troubleshoot based on your configurations. Could you please create a support case instead so we may discuss details on your resource configurations?

Please do not post any sensitive information over re:Post since this is a public platform.

As always, feel free to reach back with any further questions or concerns in the meantime!

AWS
Roopa_K
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions