Built in Chef recipe (dependencies) failing after years of running

I have an OpsWorks stack running Chef 11.10 that has been working fine for several years but recently autoscaling needed to start up a new instance and it keeps failing. Looking into it it seems the 'dependencies' built-in recipe is trying to install the OS package 'python2.7' but failing when it gets to upgrading setuptools in pip because the domain is no longer trusted.

This post seems to show what the fix is but I don't know how I can apply that to the built-in recipe: https://stackoverflow.com/a/29751768/38258

This is the portion of the startup log with the error:

\[2021-03-23T23:11:22+00:00] INFO: template\[/etc/motd.opsworks-static] backed up to /root/.chef/local-mode-cache/backup/etc/motd.opsworks-static.chef-20210323231122.368098  
\[2021-03-23T23:11:22+00:00] INFO: template\[/etc/motd.opsworks-static] updated file contents /etc/motd.opsworks-static  
\[2021-03-23T23:11:22+00:00] INFO: Processing package\[python] action install (python::package line 41)  
\[2021-03-23T23:11:22+00:00] INFO: Processing package\[python-dev] action install (python::package line 41)  
\[2021-03-23T23:11:25+00:00] INFO: Processing cookbook_file\[/var/lib/aws/opsworks/cache.stage2/get-pip.py] action create (python::pip line 36)  
\[2021-03-23T23:11:25+00:00] INFO: cookbook_file\[/var/lib/aws/opsworks/cache.stage2/get-pip.py] created file /var/lib/aws/opsworks/cache.stage2/get-pip.py  
\[2021-03-23T23:11:25+00:00] INFO: cookbook_file\[/var/lib/aws/opsworks/cache.stage2/get-pip.py] updated file contents /var/lib/aws/opsworks/cache.stage2/get-pip.py  
\[2021-03-23T23:11:25+00:00] INFO: cookbook_file\[/var/lib/aws/opsworks/cache.stage2/get-pip.py] mode changed to 644  
\[2021-03-23T23:11:25+00:00] INFO: Processing execute\[install-pip] action run (python::pip line 42)  
\[2021-03-23T23:11:26+00:00] INFO: execute\[install-pip] ran successfully  
\[2021-03-23T23:11:26+00:00] INFO: Processing python_pip\[setuptools] action upgrade (python::pip line 50)  
\[2021-03-23T23:11:26+00:00] INFO: Upgrading python_pip\[setuptools] version from uninstalled to latest  
   
================================================================================  
Error executing action `upgrade` on resource 'python_pip\[setuptools]'  
================================================================================  
   
   
Mixlib::ShellOut::ShellCommandFailed  
------------------------------------  
Expected process to exit with \[0], but received '2'  
---- Begin output of /usr/bin/pip install  --upgrade setuptools ----  
STDOUT: Downloading/unpacking setuptools from https://files.pythonhosted.org/packages/94/75/05e1d69c61c4dfaf65ad12785cd18bedc1e0129976c55914d6aea59c7da8/setuptools-54.2.0.tar.gz#sha256=aa9c24fb83a9116b8d425e53bec24c7bfdbffc313c2159f9ed036d4a6dd32d7d  
Cleaning up...  
Exception:  
Traceback (most recent call last):  
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main  
status = self.run(options, args)  
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run  
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)  
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1198, in prepare_files  
do_download,  
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1376, in unpack_url  
self.session,  
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 546, in unpack_http_url  
resp = session.get(target_url, stream=True)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 479, in get  
return self.request('GET', url, **kwargs)  
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 237, in request  
return super(PipSession, self).request(method, url, *args, **kwargs)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 467, in request  
resp = self.send(prep, **send_kwargs)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 570, in send  
r = adapter.send(request, **kwargs)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/adapters.py", line 385, in send  
raise SSLError(e)  
SSLError: hostname 'files.pythonhosted.org' doesn't match either of 'r.ssl.fastly.net', '**.catchpoint.com', '**.cnn.io', '**.dollarshaveclub.com', '**.eater.com', '**.fastly.picmonkey.com', '**.files.saymedia-content.com', '**.ft.com', '**.meetupstatic.com', '**.nfl.com', '**.pagar.me', '**.picmonkey.com', '**.realself.com', '**.sbnation.com', '**.shakr.com', '**.streamable.com', '**.surfly.com', '**.theverge.com', '**.thrillist.com', '**.vox-cdn.com', '**.vox.com', '*.voxmedia.com', 'eater.com', 'ft.com', 'i.gse.io', 'picmonkey.com', 'realself.com', 'static.wixstatic.com', 'streamable.com', 'surfly.com', 'theverge.com', 'vox-cdn.com', 'vox.com', 'www.joyent.com'  
   
Storing debug log for failure in /root/.pip/pip.log  
STDERR:   
---- End output of /usr/bin/pip install  --upgrade setuptools ----  
Ran /usr/bin/pip install  --upgrade setuptools returned 2  
   
   
Cookbook Trace:  
---------------  
/var/lib/aws/opsworks/cache.stage2/cookbooks/python/providers/pip.rb:158:in `pip_cmd'  
/var/lib/aws/opsworks/cache.stage2/cookbooks/python/providers/pip.rb:139:in `install_package'  
/var/lib/aws/opsworks/cache.stage2/cookbooks/python/providers/pip.rb:144:in `upgrade_package'  
/var/lib/aws/opsworks/cache.stage2/cookbooks/python/providers/pip.rb:60:in `block (2 levels) in class_from_file'  
/var/lib/aws/opsworks/cache.stage2/cookbooks/python/providers/pip.rb:58:in `block in class_from_file'  
   
   
Resource Declaration:  
---------------------  
# In /var/lib/aws/opsworks/cache.stage2/cookbooks/python/recipes/pip.rb  
   
50: python_pip 'setuptools' do  
51:   action :upgrade  
52:   version node\['python']\['setuptools_version']  
53: end  
   
   
   
Compiled Resource:  
------------------  
# Declared in /var/lib/aws/opsworks/cache.stage2/cookbooks/python/recipes/pip.rb:50:in `from_file'  
   
python_pip("setuptools") do  
action \[:upgrade]  
retries 0  
retry_delay 2  
cookbook_name "python"  
recipe_name "pip"  
package_name "setuptools"  
options " --upgrade"  
timeout 900  
end  
   
   
   
\[2021-03-23T23:11:27+00:00] INFO: Running queued delayed notifications before re-raising exception  
\[2021-03-23T23:11:27+00:00] ERROR: Running exception handlers  
\[2021-03-23T23:11:27+00:00] ERROR: Exception handlers complete  
\[2021-03-23T23:11:27+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out  
\[2021-03-23T23:11:27+00:00] ERROR: python_pip\[setuptools] (python::pip line 50) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with \[0], but received '2'  
---- Begin output of /usr/bin/pip install  --upgrade setuptools ----  
STDOUT: Downloading/unpacking setuptools from https://files.pythonhosted.org/packages/94/75/05e1d69c61c4dfaf65ad12785cd18bedc1e0129976c55914d6aea59c7da8/setuptools-54.2.0.tar.gz#sha256=aa9c24fb83a9116b8d425e53bec24c7bfdbffc313c2159f9ed036d4a6dd32d7d  
Cleaning up...  
Exception:  
Traceback (most recent call last):  
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main  
status = self.run(options, args)  
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run  
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)  
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1198, in prepare_files  
do_download,  
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1376, in unpack_url  
self.session,  
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 546, in unpack_http_url  
resp = session.get(target_url, stream=True)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 479, in get  
return self.request('GET', url, **kwargs)  
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 237, in request  
return super(PipSession, self).request(method, url, *args, **kwargs)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 467, in request  
resp = self.send(prep, **send_kwargs)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 570, in send  
r = adapter.send(request, **kwargs)  
File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/adapters.py", line 385, in send  
raise SSLError(e)  
SSLError: hostname 'files.pythonhosted.org' doesn't match either of 'r.ssl.fastly.net', '**.catchpoint.com', '**.cnn.io', '**.dollarshaveclub.com', '**.eater.com', '**.fastly.picmonkey.com', '**.files.saymedia-content.com', '**.ft.com', '**.meetupstatic.com', '**.nfl.com', '**.pagar.me', '**.picmonkey.com', '**.realself.com', '**.sbnation.com', '**.shakr.com', '**.streamable.com', '**.surfly.com', '**.theverge.com', '**.thrillist.com', '**.vox-cdn.com', '**.vox.com', '*.voxmedia.com', 'eater.com', 'ft.com', 'i.gse.io', 'picmonkey.com', 'realself.com', 'static.wixstatic.com', 'streamable.com', 'surfly.com', 'theverge.com', 'vox-cdn.com', 'vox.com', 'www.joyent.com'  
   
Storing debug log for failure in /root/.pip/pip.log  
STDERR:   
---- End output of /usr/bin/pip install  --upgrade setuptools ----  
Ran /usr/bin/pip install  --upgrade setuptools returned 2  
\[2021-03-23T23:11:27+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)  

Edited by: ngotch on Mar 24, 2021 6:34 AM

Edited by: ngotch on Mar 24, 2021 1:54 PM