EMR Serverless and Pandas Failing due to Numpy C-extensions

0

I've followed the methods for adding Python libraries. Documentation here: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/using-python-libraries.html

Boto installs and loads correctly at runtime (custom library working). However, EMR Serverless job fails when Pandas is loaded during runtime with the following error:

"Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed."

I've followed the instructions at: https://numpy.org/devdocs/user/troubleshooting-importerror.html, but it did not help.

I'm using Python 3.7.16 and EMR Serverless 6.10.0.

1 Answer
0

I switched to an ARM based EC2 and it worked. Specifically, I used this AMI: amzn2-ami-kernel-5.10-hvm-2.0.20230504.1-arm64-gp2.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions