I am doing some NLP work using spaCy in a ml.p2.xlarge notebook instance in sagemaker.
However, I get the following error:
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 3.10 GiB for an array with shape (2889884, 288) and data type float32
To fix the issue, I have tried the following commands:
!echo 1 | sudo tee /proc/sys/vm/overcommit_memory
!echo 1 > /proc/sys/vm/overcommit_memory
Neither of them work and they both return:
/bin/sh: /proc/sys/vm/overcommit_memory: Permission denied
Any suggestions? I even added all policies with "admin" in them to the notebook IAM with no luck.