cfn-init issue on Ubuntu 24.04 AMI

Question

Hello, I'm attempting to use the latest Ubuntu 24.04 AMI returned by

/aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id

and running into 2 issues with cfn-init.

1, what is the correct method to install aws-cfn-bootstrap on Ubuntu 24.04? I am finding this user data that worked on 20.04 is now leading to issues in 24.04 with Python 3.12 stating that "This environment is externally managed"

      UserData:
        Fn::Base64:
          !Sub |
            #!/bin/bash
            apt-get -o DPkg::Lock::Timeout=30 update
            apt-get -o DPkg::Lock::Timeout=30 install -y python3-pip
            mkdir -p /opt/aws/bin
            pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
            ln -s /usr/local/bin/cfn-* /opt/aws/bin/
            /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LinuxInstance3 --configsets setup --region ${AWS::Region}
            /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource LinuxInstance3 --region ${AWS::Region}

If I ignore the warning from Python 3.12 and use "pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz --break-system-packages" then I run into issue nr. 2 below.

2/ It seems like in Python 3.12 the 'imp' package is no longer available.

Collecting https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
  Downloading https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz (549 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 549.6/549.6 kB 1.3 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting chevron (from aws-cfn-bootstrap==2.0)
  Downloading chevron-0.14.0-py3-none-any.whl.metadata (4.9 kB)
Collecting docutils (from aws-cfn-bootstrap==2.0)
  Downloading docutils-0.21.2-py3-none-any.whl.metadata (2.8 kB)
Collecting python-daemon<2.3,>=2.2.4 (from aws-cfn-bootstrap==2.0)
  Downloading python_daemon-2.2.4-py2.py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from aws-cfn-bootstrap==2.0) (68.1.2)
Collecting lockfile>=0.10 (from python-daemon<2.3,>=2.2.4->aws-cfn-bootstrap==2.0)
  Downloading lockfile-0.12.2-py2.py3-none-any.whl.metadata (2.4 kB)
Downloading python_daemon-2.2.4-py2.py3-none-any.whl (35 kB)
Downloading chevron-0.14.0-py3-none-any.whl (11 kB)
Downloading docutils-0.21.2-py3-none-any.whl (587 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 587.4/587.4 kB 28.3 MB/s eta 0:00:00
Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Building wheels for collected packages: aws-cfn-bootstrap
  Building wheel for aws-cfn-bootstrap (setup.py): started
  Building wheel for aws-cfn-bootstrap (setup.py): finished with status 'done'
  Created wheel for aws-cfn-bootstrap: filename=aws_cfn_bootstrap-2.0-py3-none-any.whl size=635448 sha256=4d121611073517cf69e0bea758b50dbe3cc9f03e1d8ae202ef41e7c08f0a7ea9
  Stored in directory: /root/.cache/pip/wheels/2c/bb/da/f4ec6dfab336fb48ef0e3f9c469b6f469f61a1b005c518bcc2
Successfully built aws-cfn-bootstrap
Installing collected packages: lockfile, chevron, docutils, python-daemon, aws-cfn-bootstrap
Successfully installed aws-cfn-bootstrap-2.0 chevron-0.14.0 docutils-0.21.2 lockfile-0.12.2 python-daemon-2.2.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 21, in <module>
    from cfnbootstrap.cfn_client import CloudFormationClient
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/cfn_client.py", line 24, in <module>
    from cfnbootstrap import aws_client, util
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/aws_client.py", line 23, in <module>
    from cfnbootstrap import util, public_constants
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/util.py", line 17, in <module>
    from cfnbootstrap import endpoint_tool
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/endpoint_tool.py", line 16, in <module>
    from cfnbootstrap.resources import documents
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/resources/documents/__init__.py", line 17, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-signal", line 20, in <module>
    from cfnbootstrap.cfn_client import CloudFormationClient
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/cfn_client.py", line 24, in <module>
    from cfnbootstrap import aws_client, util
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/aws_client.py", line 23, in <module>
    from cfnbootstrap import util, public_constants
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/util.py", line 17, in <module>
    from cfnbootstrap import endpoint_tool
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/endpoint_tool.py", line 16, in <module>
    from cfnbootstrap.resources import documents
  File "/usr/local/lib/python3.12/dist-packages/cfnbootstrap/resources/documents/__init__.py", line 17, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'
2024-07-25 16:55:49,844 - cc_scripts_user.py[WARNING]: Failed to run module scripts_user (scripts in /var/lib/cloud/instance/scripts)
2024-07-25 16:55:49,844 - util.py[WARNING]: Running module scripts_user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 24.1.3-0ubuntu3.3 finished at Thu, 25 Jul 2024 16:55:49 +0000. Datasource DataSourceEc2Local.  Up 69.92 seconds

How can I correctly bootstrap and use cfn-init with Ubuntu 24.04?

iBehr · Accepted Answer

The issue you are hiving is not specific to Ubuntu 24.04 but rather to Python 3.12.

See the Python 3.12 Release notes:  
https://docs.python.org/3/whatsnew/3.12.html

## Important deprecations, removals or restrictions:
- PEP 623: Remove wstr from Unicode objects in Python’s C API, reducing the size of every str object by at least 8 bytes.
- PEP 632: Remove the distutils package. See the migration guide for advice replacing the APIs it provided. The third-party Setuptools package continues to provide distutils, if you still require it in Python 3.12 and beyond.
- gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.
- The asynchat, asyncore, and **imp** modules have been removed, along with several unittest.TestCase method aliases.

You should be able to follow steps such as [these](https://www.dedicatedcore.com/blog/install-pyenv-ubuntu/) to install Pyenv on Ubuntu 24.04 and then install a version of Python 3.11 or lower until you can refactor your code for Python 3.12.

Hope this helps!

Beaty · Answer

Second Ben's reply / suggestion.  This needs to be upgraded directly in the script.

Florian H · Answer

The package *aws-cfn-bootstrap* can be installed using a virtual environment for python (version 11 or lower). The following script can be used inside the Cloudformation UserData section for Ubuntu 24.04 LTS (or similar) as follows:

```
# python 3.11 virtualenv installation
add-apt-repository -y ppa:deadsnakes/ppa
apt install -y python3.11 python3-pip

# virtualenv set-up (pip binaries will use PATH=/opt/aws/virtualvenv/bin:...)
python3.11 -m pip install virtualenv
virtualenv /opt/aws/virtualvenv
source /opt/aws/virtualvenv/bin/activate

# install aws-cfn-bootstrap dependencies
pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz

# testing (no need to specify the path)
cfn-init -v --stack ${AWS::StackId} --resource EC2Instance --configsets full_install --region ${AWS::Region}
cfn-signal -e $? --stack ${AWS::StackId} --resource EC2Instance --region ${AWS::Region}
```

cfn-init issue on Ubuntu 24.04 AMI

Important deprecations, removals or restrictions:

Relevant content