Amazon Linux 2023 - issue with installing packages with cloud-init

Based on my testing with the amazon linux 2023 image and SSM enabled, I see the following processes utilizing yum during the cloud-init phase of the EC2 initialization

From ~17 to ~39 seconds rpm -U --force ./amazon-cloudwatch-agent.rpm

From ~70 to ~86 seconds rpm -U amazon-ssm-agent.rpm

I tested this using the following in my user-data file:

timeout=0
while [ -f /var/lib/rpm/.rpm.lock ];
do
  echo "$timeout seconds"
  ls -a /var/lib/rpm/
  file /var/lib/rpm/.rpm.lock
  ps -ef|grep rpm
  # When the timeout is equal to zero, show an error and leave the loop.
  if [ "$timeout" == 120 ]; then
    break
  fi
  sleep 1
  ((timeout++))
done

Instead of performing an install after a sleep or when the lock is released, you will have better luck if you just wrap your yum installs in retry logic. Here is an example:

max_attempts=5
attempt_num=1
success=false
while [ $success = false ] && [ $attempt_num -le $max_attempts ]; do
  echo "Trying yum install"
  yum update -y
  yum install java-1.8.0 java-17-amazon-corretto-devel.x86_64 wget telnet -y
  # Check the exit code of the command
  if [ $? -eq 0 ]; then
    echo "Yum install succeeded"
    success=true
  else
    echo "Attempt $attempt_num failed. Sleeping for 3 seconds and trying again..."
    sleep 3
    ((attempt_num++))
  fi
done