Setting up AWS Cloud9 with Python3.9 Runtime and urllib3

Sorry to hear, that you have been struggling for hours with upgrading python - the process can be a bit tedious.

Let me give you a bit context and sort out three independent aspects before I give you the corresponding step by step guide. Most of this is applicable to any linux distribution, altough I name commands and versions specific to Amazon Linux 2.

tl;dr -> jump to solutions directly

Context

1. Cloud 9 and AWS Lambda runtime Cloud 9 launches Amazon EC2 instances, aka virtual machines, where we have full control over the operating system. AWS Lambda and its corresponding runtimes are part of the serverless offerings of AWS, where you don't have to manage the underlying operating system. So, the suggestion by the author:" you mentiond is related but not specifically the case here.

2. Python versions The default python version in Amazon Linux 2 is Python 3.7.16 ($ python --version). The binary in fact is /usr/bin/python3 on the file system ($ which python), which is a symlink/alias to python3.7 ($ ll /usr/bin/python3). Analogously you find python2.7 (pre) installed to run "old" python scripts although python3 is the standard. If you install python 3.9 using $ sudo make altinstall it won't overwrite the default python version of the operating system. What you get is a new binary python3.9. This is kind of the lightwight approach of the operating system to support different python versions at the same time. Also the packages and pacakge versions installed are independent for each python version. I.e. you also have pip3.7, pip3.9... So, we have to be explicit in telling which python version we want to use i.e. $ python3.9 --version will return Python 3.9.16 after installation.

3. Openssl versions In terms of openssl the situation is similar. Amazon Linux 2 provides long term support (see). Therefore the default openssl version ($ openssl version) is a patched version of OpenSSL 1.0.2k. Find more details on openss versions here. Additionally a version from the openssl 1.1.1 LTS release is installed under a different binary named openssl11 ($ openssl11 version -> OpenSSL 1.1.1g). Correspondingly there exist two packages in the repository named openssl-devel and openssl11-devel, which contain files for development of applications which will use OpenSSL. A custom python installation depends on this package (see prerequisites section of your guide)

4. urrlib3 versions Version 2.0.0 of urllib3 removed support for OpenSSL versions earlier than 1.1.1. I.e. urrlib3 >=2.0.0 won't work with the default openssl version of Amazon Linux 2. This is not a problem with the default installation per se as it uses version <2.0.0 ($ pip3 list | grep urllib -> urllib3 1.26.15). However, with your custom python 3.9 version you installed/got a urrlib3 >=2.0.0 which is incompatible with OpenSSL 1.0.2k

Solutions

A: In case you need pthon 3.9 but don't require urrlib3 >=2.0.0 Prerequisite is python 3.9 installed according to your guide. $ python3.9 --version returns Python 3.9.16 downgrade urllib3 by $ pip3.9 install urllib3==1.26.15

B: In case you need pthon 3.9 and want the latest version of urrlib3 >=2.0.0 Start with a new Cloud9 environment or uninstall python 3.9 Ignore the prerequisite section of your guide and do the following instead:

$ sudo yum remove openssl-devel
$ sudo yum install gcc openssl11-devel bzip2-devel libffi-devel 

Having openssl11-devel installed instead of openssl-devel at the time when python gets installed (!) will cause the python ssl package to bind agains OpenSSL 1.1.1g, so that it is compatible with urrlib3 >=2.0.0, too.

C: Make python3 the default python version Overwrite the python alias with $ alias python='python3.9'. If needed do the same with pip $ alias pip='pip3.9'. To make it persistent add the two lines in one of your shell config files, e.g. ~/.bashrc.

Bests Norman