How do I troubleshoot high CPU usage on an Amazon EC2 Windows instance?

Short description

High CPU usage on an EC2 Windows instance might occur for the following reasons:

• You no longer have CPU credits available on your burstable instance.
• Windows Defender performed a scan that uses high CPU.
• Amazon Inspector performed a vulnerability assessment that uses high CPU.
• Amazon CloudWatch agent uses high CPU because of configuration issues.
• An application or system process uses high CPU.

To resolve high CPU usage issues, make sure that you can connect to the instance and that the instance passes its status checks. Then, identify the process that uses the CPU. Take troubleshooting actions based on the issue that you identified.

Note: Some of the following resolution steps require you to stop and start the instance.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon Elastic Block Store (Amazon EBS) volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your Amazon EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instances don't terminate when you stop them.

Make sure that you can connect to the instance

Use Remote Desktop Protocol (RDP) to connect to the instance. If you can't connect with RDP because of the high CPU usage, then start a session with Session Manager, a capability of AWS Systems Manager.

If you can't use Session Manager, then temporarily change the instance type to a larger size. Then, connect with RDP.

If you still can't connect, then complete the following steps to resolve your connection issues:

1. Launch a rescue instance in the same operating system.
2. Stop the original instance.
3. Detach the original instance's root Amazon Elastic Block Store (Amazon EBS) volume.
4. Attach the Amazon EBS volume to the rescue instance.
5. Run EC2Rescue in the Offline instance mode with the FixAll command.
6. Detach the EBS volume from the rescue instance.
7. Attach the volume to the original instance as the root volume.
8. Start the original instance.

Make sure that your instance passes its status checks

Complete the following steps:

1. Open the Amazon EC2 console.
2. Choose Instances, and then select your instance.
3. Choose the Status checks tab, and then make sure that the status for both status checks is Passed.

To troubleshoot instance status check failures, see How do I troubleshoot status check failures on my Amazon EC2 Windows instance?

Identify the processes that cause the high CPU usage

Use AWSSupport-TroubleshootWindowsPerformance to check for high CPU processes

To automatically identify processes that use high CPU, run the AWSSupport-TroubleshootWindowsPerformance runbook.

Use CloudWatch to identify the cause of high CPU

Complete the following steps:

1. Open the Amazon CloudWatch console.
2. In the navigation pane, choose Metrics, and then choose EC2.
3. Choose Per-Instance Metrics.
4. Choose CPUUtilization, and then filter for the past 24-72 hours.
5. In the graph, check for the following factors:
   Sustained high CPU that occurs because of a persistent process or resource constraint.
   Periodic spikes at regular intervals that occur because of scheduled tasks such as Amazon Inspector scans that run every 6 hours, or Windows Defender scans.
   CPUUtilization that's stuck at a value that's lower than 100%. This issue occurs if your burstable instance has no more CPU credits available, so the instance operates at the baseline usage.

Check Windows to identify the processes the cause high CPU usage

Use RDP to connect to your instance,and then run the following PowerShell command:

 Get-Process | Sort-Object CPU -Descending |
  Select-Object -First 10 Name, Id, CPU, @{N='WS(MB)';E={[math]::Round($_.WorkingSet64/1MB,1)}} |
  Format-Table -AutoSize

Or, use RDP to connect, and then choose the Start menu. Open the Task Manager, and then choose the Details tab. Sort the results by CPU to view tasks that use the highest CPU.

If you can't connect with RDP, then complete the following steps to use Run Command, a capability of AWS Systems Manager, to check CPU usage:

1. Open the AWS Systems Manager console.
2. In the navigation pane, choose Run Command.
3. Choose AWS-RunPowerShellScripts, and then enter the following command:

    Get-Process | Sort-Object CPU -Descending |
     Select-Object -First 10 Name, Id, CPU, @{N='WS(MB)';E={[math]::Round($_.WorkingSet64/1MB,1)}} |
     Format-Table -AutoSize

Based on the process that you identified, take the following actions:

• If CPU throttles on a T-family instance, then troubleshoot CPU credit exhaustion.
• If MsMpEng.exe uses 30-100% of the CPU, then resolve issues with Windows Defender processes.
• If WmiPrvSE.exe or InspectorSsmPlugin.exe uses at least 25% of the CPU, then resolve issues with Amazon Inspector processes.
• If amazon-cloudwatch-agent.exe uses 50-90% of the CPU, then resolve issues with CloudWatch agent.
• If an application or Windows system process uses high CPU, then resolve issues with that process.

Troubleshoot CPU credit exhaustion on burstable instances

If you use a burstable instance type such as T2, T3, T3a, or T4g, then CPU credit exhaustion might throttle your performance. To resolve this issue, see How can I resolve high CPU utilization on my T2 or T3 EC2 Windows instance if my CPU is being throttled?

Troubleshoot high Windows Defender CPU usage

If the Windows Defender Scheduled Scan in Task Scheduler shows a Last Run Time of November 30, 1999, then there might be issues with Windows Defender. You can also check for the "0x00000003" error code in the Windows Defender operation log.

Identify Windows Defender issues

To find the logs, complete the following steps:

1. Choose the Start menu, and then open Application and Service Logs.
2. Choose Microsoft, and then choose Windows.
3. Choose Windows Defender, and then choose Operational.

If you experience high CPU usage after system updates, instance restarts, or Amazon EBS volume attachment, then the issue might be Windows Defender. To resolve Windows Defender issues, take the following actions.

Stop and start the instance

Stop and start your instance.

Ater you instance restarts, choose the Start menu, and then open Task Scheduler. Make sure that the Windows Defender Scheduled Scan shows a recent Last Run Time.

Reduce the CPU effect of scheduled scans

Complete the following steps:

1. Choose the Start menu.
2. Open Task Scheduler, and then choose Task Scheduler Library.
3. Choose Microsoft, and then choose Windows.
4. Choose Windows Defender.
5. Double-click Windows Defender Scheduled Scan, and then configure the following settings:
   Choose the General tab, and then clear Run with highest privileges.
   Choose the Conditions tab, and then clear all options.
   Choose the Settings tab, and then set Stop the task if it runs longer than to 1 hour.

Exclude AWS agent paths from scans

Run the following command to exclude the AWS agents from scans:

Set-MpPreference -ExclusionPath @(
    "C:\Program Files\Amazon",
    "C:\ProgramData\Amazon"
)
Set-MpPreference -ExclusionProcess @(
    "C:\Program Files\Amazon\SSM\amazon-ssm-agent.exe",
    "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.exe"
)

Note: You can also add the D:\MyApp\Logs.ext application directories to the exclusion list to the preceding command.

To verify the paths that you excluded, run the following command:

Get-MpPreference | Select-Object -ExpandProperty ExclusionPath
Get-MpPreference | Select-Object -ExpandProperty ExclusionProcess

Update your Windows Defender antivirus definitions

Run the following command:

Update-MpSignature
Get-MpComputerStatus | Select-Object AntivirusSignatureLastUpdated, AntivirusSignatureVersion

Troubleshoot high Amazon Inspector CPU usage

By default, Amazon Inspector scans Windows instances every 6 hours with Windows Management Instrumentation (WMI) queries that use high CPU. To reduce the effect of these scans, take one or more of the following actions.

• Set a custom scan frequency for your Windows instance.
• Change the Amazon Inspector scan mode to Agentless scanning.
  Note: With agentless scanning, Amazon Inspector uses less CPU because it scans with Amazon EBS snapshots instead of WMI queries. Agentless scanning runs every 24 hours.
• Exclude your instance from the Amazon Inspector scan.
  Important: To exclude your instance, you must allow access to tags in your instance metadata.

Troubleshoot high CloudWatch agent CPU usage

If the high CPU usage occurs when you start the agent or load the CloudWatch agent configuration, then you must resolve CloudWatch agent issues. To troubleshoot CloudWatch agent CPU usage issues, see How do I troubleshoot high CPUUtilization from the unified CloudWatch agent on my server?

Monitor your CPU usage

To proactively identify potential issues, use CloudWatch to monitor the CPU usage of your instances.

Create CloudWatch alarms for the following conditions:

• CPUUtilization is greater than 80% for 10 consecutive minutes.
• (Burstable instances only) CPUCreditBalance is less than 50 credits.
• StatusCheckFailed is greater than or equal to 1.
  Note: For this metric, also add a recover action.

You can also use the procstat plugin to monitor processes that use high CPU usage.

Also, keep your instance's drivers updated to the latest versions.

Related information

How do I diagnose high CPU utilization on my EC2 Windows instance when my CPU doesn't throttle?

How do I troubleshoot high CPU utilization on an Amazon EC2 Linux instance?