Why does my Amazon EC2 Linux instance automatically terminate?

Resolution

An Amazon EC2 Linux instance terminates because of encrypted Amazon Block Store (Amazon EBS) volumes or because the audit daemon configuration has low disk space.

The instance terminates because of encrypted EBS volumes

To check if a volume is encrypted, open the Amazon EC2 console, and then select Volumes. Volumes that are encrypted with an AWS Key Management Service (AWS KMS) key show the label Encrypted in the Encryption column.

Review the system logs to check if the instance started to boot the operating system (OS). If the instance didn't boot the OS and the volumes are encrypted, then the AWS Identity and Access Management (IAM) role doesn't have AWS KMS key access permissions.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To check the IAM permissions, complete the following steps:

1. Run the describe-instances command to verify the StateReason error message and error code:

   $ aws ec2 describe-instances --instance-id i-example-id --region example-region --query "Reservations[].Instances[].{StateReason:StateReason}" --output json

   Note: Replace i-example-ID with your instance ID. Replace example-region with your AWS Region.
   If an encrypted volume that's attached to the instance has permissions or policy issues, then you receive a client error. You see an output that's similar to the following message:

   [ 
    [
      {
    "StateReason": {
    "Code": "Client.InternalError",
    "Message": "Client.InternalError: Client error on launch"
      }
    }
    ]
   ]

2. Verify that the IAM role has the correct IAM permissions:

   {
       "Version": "2012-10-17",
       "Statement": {
           "Effect": "Allow",
           "Action": [
               "kms:CreateGrant",
               "kms:Decrypt",
               "kms:DescribeKey",
               "kms:GenerateDataKeyWithoutPlainText",
               "kms:ReEncrypt"
           ],
           "Resource": [
               "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
               "arn:aws:kms:us-west-2:111122223333:key/0987dcba-09fe-87dc-65ba-ab0987654321"
           ]
       }
   }
   

If you launched the instance indirectly through another service, such as Amazon EC2 Auto Scaling, then configure the key policy with the correct permissions.

The audit daemon configuration terminates instance because of a disk full error

If the instance started booting, then access the boot logs in an instance screenshot or system logs. You see an output that's similar to the following message:

Aug 19 04:11:02 ip-10-0-202-230 auditd[4300]: Audit daemon is low on disk space for logging
Aug 19 04:11:02 ip-10-0-202-230 auditd[4300]: The audit daemon is now halting the system
Aug 19 04:11:02 ip-10-0-202-230 systemd: Stopping Session 29 of user splunk.
Aug 19 04:11:02 ip-10-0-202-230 systemd: Stopped target Timers.

To resolve disk full errors, complete the following steps:

1. Open the Amazon EC2 console.

2. In the navigation pane, choose Instances, and then select the instance.

3. Stop the instance, and then detach the EBS root volume (/dev/xvda for Linux) from the instance.

4. Use the Linux instance that's in your Availability Zone. Or, launch a new instance to use as your rescue instance.

5. Attach the EBS root volume to the rescue instance as a secondary device, for example /dev/xvdf.

6. Use SSH to connect to your rescue instance.

7. Create a mount point directory for the new root volume that you attached to the rescue instance. The following sudo commands apply only to Linux environments:

    sudo mkdir /mnt/rescue
    sudo mount /dev/xvdf /mnt/rescue

   Note: Replace /dev/xvdf with the secondary device name and /mnt/rescue with the name of your mount point directory.

8. Increase the root volume size and extend the partition.

9. Run the unmount command to unmount the volume:

   sudo umount /mnt/rescue

10. Detach the volume from the rescue instance and attach the volume to the original instance with the device name, for example /dev/xvda for Linux. Then, start the instance to confirm that it successfully boots.

11. Review the audit service configuration to make sure that it doesn't have the option to terminate the machine if it runs out of space. For Amazon Linux, Amazon Linux 2, and Amazon Linux 2023, make sure that the audit service configuration includes the following options in /etc/audit/auditd.conf:

    max_log_file_action = ROTATE
    admin_space_left_action = SUSPEND
    disk_full_action = SUSPEND
    disk_error_action = SUSPEND

Related information

Troubleshoot Amazon EC2 instance launch issues

Why can't I start or launch my EC2 instance?