Skip to content

Idle Windows Server 2025 instances: reproducible high CPU due to interrupt processing; AWS driver incompatibility?

0

The following findings can be reproduced using the attached CloudFormation stack, creating a clean t3a.micro EC2 instance running either Windows Server Core 2022 or 2025, with everything on default and nothing installed except except preinstalled AWS services like SSM.

Windows Server 2025 is burning about 10% just for interrupt processing with little to no I/O, while Windows Server 2022 does not have this problem.

I initially noticed this behavior on Elastic Beanstalk environments when switching from Windows Server 2022 to 2025. I was able to confirm this across multiple 10+ deployments in different AZ, so it's not just due to a faulty host or something like that. It might also affect RDS SQL Server.

Interestingly, from within the instance, Windows reports comparable % processor utility for both 2022 and 2025; way less than what EC2 is reporting for 2025 instances. % Processor Time though is much higher and matches what EC2 is reporting; the difference is explained by % Interrupt Time which also corresponds to a higher rate of interrupts.

The process list is not very different, and no process has significant CPU usage, Idle is always > 95%.

With just a SSM Session Manager session open with an idle PowerShell window (hours after initial deployments, so it isn't affected by initial Defender runs etc.), EC2 reports ~6% for Win2022 and ~12% for Win2025. This also means that a t3a.micro instance is collecting CPU credit balance with Win2022, while it stays flat at 0 with Win2025 because it is constantly above the 10% baseline performance.

On the 2025 instance

Querying relevant Windows Performance counters:

Get-Counter '\Processor Information(_Total)\% Processor Utility', '\Processor(_Total)\% Processor Time', '\Processor(_Total)\% DPC Time', '\Processor(_Total)\% Interrupt Time', '\Processor(_Total)\Interrupts/sec', '\Network Interface(*)\Bytes Total/sec', '\PhysicalDisk(_Total)\Disk Transfers/sec'

Result:

Timestamp                  CounterSamples
---------                  --------------
7/21/2026 1:32:09 PM       \\ec2amaz-oit0diu\processor information(_total)\% processor utility :
                           3.34730170794957

                           \\ec2amaz-oit0diu\processor(_total)\% processor time :
                           11.9038312409459

                           \\ec2amaz-oit0diu\processor(_total)\% dpc time :
                           0

                           \\ec2amaz-oit0diu\processor(_total)\% interrupt time :
                           10.9145695807678

                           \\ec2amaz-oit0diu\processor(_total)\interrupts/sec :
                           1562.77225019445

                           \\ec2amaz-oit0diu\network interface(amazon elastic network adapter)\bytes total/sec :
                           0

                           \\ec2amaz-oit0diu\physicaldisk(_total)\disk transfers/sec :
                           2.99381657125373

As you can see, there is a high rate of interrupts, while disk and network I/O is low.

OS and AWS driver versions:

(Get-WmiObject -class Win32_OperatingSystem).Caption; driverquery /v | findstr /i "ena aws nvme"

Result:

Microsoft Windows Server 2025 Datacenter

AWSNVMe      AWSNVMe                AWSNVMe                Kernel        Boot       Running    OK         TRUE        FALSE        4,096             135,168     0                                 C:\Windows\system32\drivers\AWSNVMe.sys          4,096
ena          Amazon Elastic Network Amazon Elastic Network Kernel        Manual     Running    OK         TRUE        FALSE        4,096             131,072     0          7/16/2025 11:08:19 AM  C:\Windows\system32\drivers\ena.sys              8,192
nvmedisk     Microsoft NVMe disk dr Microsoft NVMe disk dr Kernel        Boot       Running    OK         TRUE        FALSE        20,480            32,768      0                                 C:\Windows\system32\drivers\nvmedisk.sys         4,096
stornvme     Microsoft Standard NVM Microsoft Standard NVM Kernel        Manual     Running    OK         TRUE        FALSE        4,096             225,280     0                                 C:\Windows\system32\drivers\stornvme.sys         4,096
stornvmeofi  Microsoft NVMeoF Initi Microsoft NVMeoF Initi Kernel        Manual     Stopped    OK         FALSE       FALSE        8,192             65,536      0                                 C:\Windows\system32\drivers\stornvmeofi.sys      4,096

On the 2022 instance for comparison

You can see that % processor utility is similar to the 2025 instance, but % Processor time also has a similar value (roughly matching of what EC2 metrics are showing), and Interrupts/sec is lower:

Timestamp                  CounterSamples
---------                  --------------
7/21/2026 1:32:21 PM       \\ec2amaz-05iv3fa\processor information(_total)\% processor utility :
                           4.20887101691597

                           \\ec2amaz-05iv3fa\processor(_total)\% processor time :
                           1.69080333257067

                           \\ec2amaz-05iv3fa\processor(_total)\% dpc time :
                           0

                           \\ec2amaz-05iv3fa\processor(_total)\% interrupt time :
                           0

                           \\ec2amaz-05iv3fa\processor(_total)\interrupts/sec :
                           909.555213461177

                           \\ec2amaz-05iv3fa\network interface(amazon elastic network adapter)\bytes total/sec :
                           719.019942725098

                           \\ec2amaz-05iv3fa\physicaldisk(_total)\disk transfers/sec :
                           0

Versions:

Microsoft Windows Server 2022 Datacenter
AWSNVMe      AWSNVMe                AWSNVMe                Kernel        Boot       Running    OK         TRUE        FALSE        4,096             135,168     0                                 C:\Windows\system32\drivers\AWSNVMe.sys          4,096
ena          Amazon Elastic Network Amazon Elastic Network Kernel        Manual     Running    OK         TRUE        FALSE        4,096             131,072     0          7/16/2025 11:08:19 AM  C:\Windows\system32\drivers\ena.sys              8,192
stornvme     Microsoft Standard NVM Microsoft Standard NVM Kernel        Manual     Running    OK         TRUE        FALSE        4,096             151,552     0                                 C:\Windows\system32\drivers\stornvme.sys         4,096
xenbus       AWS PV Bus             AWS PV Bus             Kernel        Manual     Running    OK         TRUE        FALSE        4,096             135,168     0          5/8/2025 11:38:16 PM   C:\Windows\system32\drivers\xenbus.sys           4,096
xenfilt      AWS Bus Filter         AWS Bus Filter         Kernel        Boot       Running    OK         TRUE        FALSE        4,096             32,768      0          5/8/2025 11:38:18 PM   C:\Windows\system32\drivers\xenfilt.sys          4,096
xeniface     AWS Interface          AWS Interface          Kernel        Manual     Stopped    OK         FALSE       FALSE        4,096             65,536      0          5/8/2025 10:23:04 PM   C:\Windows\system32\drivers\xeniface.sys         4,096
xenvbd       AWS PV Storage Host Ad AWS PV Storage Host Ad Kernel        Manual     Running    OK         TRUE        FALSE        0                 90,112      0          9/18/2024 9:49:51 PM   C:\Windows\system32\drivers\xenvbd.sys           4,096
xenvif       AWS PV Network Class   AWS PV Network Class   Kernel        Manual     Stopped    OK         FALSE       FALSE        4,096             241,664     0          9/27/2024 8:54:27 PM   C:\Windows\system32\drivers\xenvif.sys           4,096

What can we do to resolve this issue?

asked 23 days ago130 views

5 Answers
5
Accepted Answer

By running these checks, you have ruled out ENA driver misconfiguration and forced synthetic timers.

However, your bcdedit output reveals a fundamental architectural divergence between your two deployments:

  • WS2022 is booting via \Windows\system32\winload.exe (Legacy BIOS).
  • WS2025 is booting via \EFI\Microsoft\Boot\bootmgfw.efi and winload.efi (UEFI).

My understanding is, that this is a critical distinction. When a Windows instance boots in UEFI mode on the AWS Nitro system, the Hardware Abstraction Layer (HAL) interacts differently with the hypervisor compared to Legacy BIOS. Specifically, ACPI table parsing, timer routing (e.g., HPET, RTC, synthetic timers), and the interrupt controller initialization (APIC/x2APIC) are handled via entirely different execution paths.

The baseline interrupt storm (~1,500 interrupts/sec) you are observing is almost certainly a symptom of how the WS2025 kernel/HAL is mapping or polling hardware interrupts in this specific UEFI environment on the underlying AMD EPYC (t3a) architecture.

To answer your question directly: Yes, a Windows Performance Recorder (WPR) trace is exactly the correct next step. As far as I know, it's the only way to move from hypothesis to absolute certainty by proving exactly which driver or system routine is triggering the interrupts.

How to proceed with the WPR Trace

1. Capture the Trace Open an elevated command prompt on the idle WS2025 instance and run:

wpr -start cpu -start Interrupt

Let the instance sit completely idle for 30–60 seconds, then stop and save the trace:

wpr -stop C:\high_interrupts.etl

2. Analyze the Trace (Locally) If you open high_interrupts.etl using Windows Performance Analyzer (WPA), expand the Computation graph and view DPC/ISR or CPU Usage (Sampled). Group the columns by Module Name and Function Name. The module consuming the CPU cycles (e.g., hal.dll, ntoskrnl.exe, ACPI.sys, or a specific Nitro driver) will immediately float to the top.

3. Escalation and Security Warning Please do not upload the raw .etl file here on re:Post. WPR traces contain deep system memory pools, file paths, and potentially sensitive background data.

Because this is a low-level OS/Hypervisor interaction issue, you need to open an AWS Technical Support case. Provide them with:

  • Your reproduction steps (WS2025/UEFI vs. WS2022/Legacy on t3a.micro).
  • The .etl trace file.

This will provide the EC2 Windows Engineering team with the exact stack trace they need to either calibrate the WS2025 AMI or patch the respective Nitro driver.

If you do open the trace in WPA, please feel free to share just the name of the faulting module here in the thread! It would be highly valuable for the community to know exactly what is causing the spike.

see also:

Evidence for the Legacy BIOS vs. UEFI difference (winload.exe vs. winload.efi) AWS clearly documents that EC2 instances (especially on the Nitro system) boot in either Legacy BIOS mode or UEFI mode depending on the AMI. Microsoft confirms that the Windows bootloader path explicitly reveals which mode the OS is running in:

Microsoft Docs: BCD System Store Settings for UEFI (Confirms that \Windows\system32\winload.efi is used exclusively for UEFI systems, whereas legacy systems use winload.exe.) -> https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/bcd-system-store-settings-for-uefi

AWS Docs: Boot modes for Amazon EC2 instances (Explains how the Nitro system provides different underlying firmware interfaces for UEFI and Legacy BIOS.) -> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ami-boot.html

Evidence for altered Interrupt & Timer routing under UEFI Under UEFI, the way the operating system's Hardware Abstraction Layer (HAL) interacts with hardware timers (such as HPET or invariant TSC) and the Advanced Configuration and Power Interface (ACPI) fundamentally changes. This handoff between the hypervisor firmware and the OS dictates how interrupts are routed (e.g., APIC vs. x2APIC), which directly impacts virtualized CPU overhead:

Microsoft Docs: Analyze High CPU Usage with WPA *(Documents the exact workflow for recording interrupt storms using wpr -start cpu and how to visualize DPC/ISR spikes down to the exact .sys driver level in the Windows Performance Analyzer). * -> https://learn.microsoft.com/en-us/windows-hardware/test/wpt/cpu-analysis

EXPERT

answered 22 days ago

  • I guess it makes sense to share the relevant part of the response I got:

    This is a confirmed known issue affecting Windows Server 2025 on AMD-based Nitro instances (not limited to t3a it affects other AMD instance families as well). The issue does not occur on Intel-based Nitro instances or on Windows Server 2022 running on the same AMD hardware.

    The root cause has been identified as an interaction between the Windows Server 2025 Hyper-V synthetic timer/clock source enlightenments and the AMD Nitro platform, resulting in an elevated interrupt rate per vCPU at idle. Our service team has validated and reproduced the behavior and has escalated it to the platform team for resolution.

    I want to set expectations that we do not currently have a timeline for when the fix will be delivered, and unfortunately we are unable to share internal development roadmaps. However, this is actively being worked on by the platform team. Once a fix is available it would be delivered through an updated AMI.

    I fully understand that problems like this might be difficult to resolve, but it's very unfortunate that this isn't documented anywhere publicly, and as a customer you have to burn hours to analyze the problem and need paid support to get confirmation that it is a known AWS problem...

    Once I can confirm that the problem has been resolved, I will add an update here.

3

I think the generic SSM runbooks mentioned above won't help here. A high % Interrupt Time (~11%) on a clean, idle OS indicates a low-level issue with kernel timer handling or hypervisor clock synchronization in Windows Server 2025 on Nitro (t3a / AMD EPYC).

To isolate the root cause, test these specific configurations on a fresh test instance:

  1. Disable Dynamic Ticks

Windows Server 2025 uses dynamic ticking to adjust timer resolution on the fly. On virtualized hardware, miscommunication between guest timer interrupts and the hypervisor frequently causes an interrupt storm. bcdedit /set disabledynamictick yes

(Requires a system reboot).

  1. Verify ENA Interrupt Moderation

Check if the Amazon Elastic Network Adapter driver in WS2025 has interrupt moderation enabled by default: Get-NetAdapterAdvancedProperty -Name "Ethernet" | Where-Object { $_.DisplayName -like "Interrupt" }

If moderation is disabled or misconfigured, small background packets can saturate a vCPU with hardware interrupts.

  1. Compare BCD Clock Settings

Run bcdedit /enum on both WS2022 and WS2025. Verify if useplatformclock or useplatformtick is enabled on the WS2025 AMI, which forces the OS to use synthetic clock sources instead of invariant TSC, causing higher overhead.

If disabling dynamic ticks drops % Interrupt Time back to WS2022 baseline levels, attach your before/after Get-Counter logs to an AWS Support case so this can be escalated to the EC2 Windows Engineering team for AMI timer calibration.

See also:

EXPERT

answered 23 days ago

EXPERT

reviewed 23 days ago

2

Thank you, I opened a case.

I also found out that this does NOT happen with non-burstable instance types like c7a!

WPA analysis attributes almost all ISR CPU time to ntoskrnl!HalpVpptAcknowledgeInterrupt, with the call stack dominated by:

ntoskrnl!KiIdleLoop → ntoskrnl!PoIdle → ntoskrnl!PpmIdleExecuteTransition → ntoskrnl!HalpVpptAcknowledgeInterrupt

Very little ISR time is attributed to ENA, AWSNVMe or Storport.

answered 22 days ago

EXPERT

reviewed 22 days ago

0

Based on your detailed analysis, you're experiencing high interrupt processing on Windows Server 2025 instances that doesn't occur on Windows Server 2022. The approximately 10% CPU usage from interrupt processing on idle instances is concerning, especially for burstable instances like t3a.micro that can't accumulate CPU credits when consistently above their baseline.

To troubleshoot and potentially resolve this issue:

Immediate Troubleshooting Steps:

  1. Run the AWSSupport-TroubleshootWindowsPerformance runbook through AWS Systems Manager. This can automatically identify processes causing high CPU usage and may provide additional insights into the interrupt processing behavior.

  2. Check for driver compatibility issues by running the AWSSupport-UpgradeWindowsAWSDrivers runbook. While your driver versions appear current, Windows Server 2025 is relatively new, and there may be compatibility improvements or updates available specifically for this OS version.

  3. Monitor with CloudWatch by setting up detailed monitoring and creating alarms for CPUUtilization. Use the procstat plugin to get more granular process-level data that might reveal patterns not visible in standard metrics.

Potential Root Causes:

Given that Windows Server 2025 is a newer operating system, there may be:

  • Compatibility issues between the OS kernel's interrupt handling and the current AWS drivers (ENA, NVMe)
  • Changes in how Windows Server 2025 processes hardware interrupts compared to 2022
  • Differences in default system services or background tasks that generate more interrupt activity

Recommended Actions:

  1. Ensure all instance drivers are fully updated and compatible with Windows Server 2025
  2. Consider temporarily using a larger instance type (beyond t3a.micro) to determine if the issue scales proportionally or if it's a fixed overhead
  3. Review your AWS Support plan options, as this appears to be a systemic compatibility issue that may require deeper investigation by AWS engineering teams
  4. Document your findings and add correspondence through the AWS Support Center at https://console.aws.amazon.com/support/ with your detailed performance counter data

Since this is reproducible across multiple deployments and availability zones, it's unlikely to be a host-specific issue and more likely represents a broader compatibility consideration with Windows Server 2025 on AWS infrastructure.
Sources
Troubleshoot high CPU usage in my EC2 Windows instances | AWS re:Post
Troubleshoot EC2 Windows Nitro instance status check failures | AWS re:Post

answered 23 days ago

0

Thank you @Florian Turnwald for your super quick and helpful answer!

Output for (2) on Server 2025:

Empty. Output without the filter:

Name                      DisplayName                    DisplayValue                   RegistryKeyword RegistryValue
----                      -----------                    ------------                   --------------- -------------
Ethernet                  IPv4 Checksum offload          Rx & Tx Enabled                *IPChecksumO... {3}
Ethernet                  Jumbo Packet                   Disabled                       *JumboPacket    {1514}
Ethernet                  Large Send Offload V1 (IPv4)   Enabled                        *LsoV1IPv4      {1}
Ethernet                  Large Send Offload V2 (IPv4)   Enabled                        *LsoV2IPv4      {1}
Ethernet                  Large Send Offload V2 (IPv6)   Enabled                        *LsoV2IPv6      {1}
Ethernet                  Maximum Number of RSS Queues   32                             *NumRssQueues   {32}
Ethernet                  Priority and Vlan Support      Both                           *PriorityVLA... {3}
Ethernet                  Receive Buffers                1024                           *ReceiveBuffers {1024}
Ethernet                  Receive Side Scaling           Enabled                        *RSS            {1}
Ethernet                  TCP Checksum Offload (IPv4)    Rx & Tx Enabled                *TCPChecksum... {3}
Ethernet                  TCP Checksum Offload (IPv6)    Rx & Tx Enabled                *TCPChecksum... {3}
Ethernet                  Transmit Buffers               1024                           *TransmitBuf... {1024}
Ethernet                  UDP Checksum Offload (IPv4)    Rx & Tx Enabled                *UDPChecksum... {3}
Ethernet                  UDP Checksum Offload (IPv6)    Rx & Tx Enabled                *UDPChecksum... {3}
Ethernet                  LLQ Header Size Policy         Recommended                    LlqPolicy       {0}
Ethernet                  Metrics Refresh Interval       1                              MetricsRefre... {1}
Ethernet                  MAC Address                    --                             NetworkAddress  {--}
Ethernet                  VLAN ID                        0                              VlanId          {0}

Same on Server 2022.

Output for (3)

On Server 2025:

Windows Boot Manager
--------------------
identifier              {bootmgr}
device                  partition=\Device\HarddiskVolume2
path                    \EFI\Microsoft\Boot\bootmgfw.efi
description             Windows Boot Manager
locale                  en-US
inherit                 {globalsettings}
bootshutdowndisabled    Yes
default                 {current}
resumeobject            {06abedb4-730f-11f0-bb11-0afff7ff4a31}
displayorder            {current}
toolsdisplayorder       {memdiag}
timeout                 30

Windows Boot Loader
-------------------
identifier              {current}
device                  partition=C:
path                    \Windows\system32\winload.efi
description             Windows Server
locale                  en-US
inherit                 {bootloadersettings}
recoverysequence        {5e607f64-734b-11f0-aaf4-0affe412849f}
displaymessageoverride  Recovery
recoveryenabled         Yes
isolatedcontext         Yes
allowedinmemorysettings 0x15000075
osdevice                partition=C:
systemroot              \Windows
resumeobject            {06abedb4-730f-11f0-bb11-0afff7ff4a31}
nx                      OptOut
bootstatuspolicy        IgnoreAllFailures

useplatformclock or useplatformtick not mentioned!

Neither on Server 2022, although the output is slighty different:

Windows Boot Manager
--------------------
identifier              {bootmgr}
device                  partition=C:
description             Windows Boot Manager
locale                  en-US
inherit                 {globalsettings}
bootshutdowndisabled    Yes
default                 {current}
resumeobject            {4426d739-00bc-11ec-a967-0e221fdbf186}
displayorder            {current}
toolsdisplayorder       {memdiag}
timeout                 30

Windows Boot Loader
-------------------
identifier              {current}
device                  partition=C:
path                    \Windows\system32\winload.exe
description             Windows Server
locale                  en-US
inherit                 {bootloadersettings}
recoverysequence        {abb264d7-85c0-11ef-bb37-0afff4384355}
displaymessageoverride  Recovery
recoveryenabled         Yes
allowedinmemorysettings 0x15000075
osdevice                partition=C:
systemroot              \Windows
resumeobject            {4426d739-00bc-11ec-a967-0e221fdbf186}
nx                      OptOut
bootstatuspolicy        IgnoreAllFailures

Disable Timer Ticks

I ran the suggested command and rebooted. Unfortunately, it didn't change anything except the output of bcdedit /enum which now includes

disabledynamictick Yes

Any ideas? Should I provide you with a WPR trace (CPU profile)?

answered 23 days ago

EXPERT

reviewed 22 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.