Default EMR Spark python environment contains two different versions of the dateutil package

Question

I noticed that when you create a new EMR cluster using Spark, the default Python environment includes two different packages that both provide the "dateutil" package:

```
py-dateutil==2.2
python-dateutil==2.8.1
```

The latter seems like the correct/official dateutil import: https://pypi.org/project/python-dateutil/ 
The former has not been updated since 2014 and has a single release: https://pypi.org/project/py-dateutil/

Is this intentional? If not, can py-dateutil be removed from the default python environment? The overlap leads to confusing situations like the linked issue, where you think you have one version of dateutil installed but actually have an older version installed: https://github.com/dagster-io/dagster/issues/22586

To reproduce, simply create a brand new EMR cluster with Spark, SSH into it, and run pip freeze.

Yokesh NK · Answer

Hello,

py-dateutil is getting installed from EMR side as part of provisioning. Under /usr/local/lib/python3.6/site-packages, you can see the file "py_dateutil-2.2.egg-info". Precedence is given to this pre-installed file at the time of puppet deployment. More details provided [**here**](https://repost.aws/knowledge-center/emr-install-python-libraries)

We do not recommend changing these package versions, as these combinations have not been tested, and we cannot predict what incompatibilities may arise with the services running in EMR. However, if you are willing to perform rigorous testing on your end to ensure everything is working correctly, you can proceed with caution.

Default EMR Spark python environment contains two different versions of the dateutil package

Relevant content