High CPU Utilization on Windows Core Instance with Kubernetes Pods Every Six Hours

Question

Hello AWS Community,

I have a Windows Core instance running Kubernetes pods, and I'm experiencing high CPU utilization spikes every six hours. Since I don't have a GUI, I need to diagnose and resolve this issue using command-line tools.

Could someone guide me on how to identify the cause of these periodic CPU spikes?

Oleksii Bebych · Answer

Diagnosing high CPU utilization on a Windows Core instance running Kubernetes pods requires a systematic approach to identify the root cause. Here's a step-by-step guide using command-line tools:

**Step 1: Monitor CPU Usage**

Using `typeperf`

`typeperf` is a command-line utility that reports performance data.

1. Open a command prompt on your Windows Core instance.
2. Use `typeperf` to log CPU utilization data over time.
```
typeperf "\Processor(_Total)\% Processor Time" -si 1 -sc 3600 -f CSV -o cpu_usage.csv
```
This command logs the CPU usage every second for an hour. Adjust -sc (sample count) as needed.

**Step 2: Identify High CPU Processes**

Using `tasklist` and `taskkill`

1. Use `tasklist` to get a list of running processes and their CPU usage.
```
tasklist /v /fo csv > tasklist.csv
```

The /v parameter provides verbose information, including CPU time.

2. To check CPU usage at a specific point, you can use `wmic`:
```
wmic path Win32_PerfFormattedData_PerfProc_Process get Name,PercentProcessorTime,IDProcess /format:csv > process_cpu_usage.csv
```

**Step 3: Analyze Kubernetes Pods**

Since you're running Kubernetes, you should check which pods are consuming resources.

1. Use `kubectl` to get a list of pods and their resource usage:
```
kubectl top pods -A
```
This command shows the CPU and memory usage for all pods across all namespaces.

2. If you identify a pod with high CPU usage, describe the pod to get more details:
```
kubectl describe pod  -n 
```

3. Check the logs of the problematic pod to see if there are any clues:
```
kubectl logs  -n 
```

**Step 4: Scheduled Tasks or Cron Jobs**

High CPU utilization spikes every six hours might be due to scheduled tasks or cron jobs.

1. List all scheduled tasks:
```
schtasks /query /fo LIST /v > scheduled_tasks.txt
```

2. Check for any cron jobs scheduled in your Kubernetes cluster:
```
kubectl get cronjobs -A
```

**Step 5: Analyze Network Activity**

Sometimes, network activity can cause high CPU usage.

1. Use `netstat` to check for active network connections:
```
netstat -ano > netstat.txt
```

2. Cross-reference the PID in the `netstat` output with your `tasklist` output to identify which processes are making network connections.

**Step 6: Use Windows Performance Recorder (WPR)**

For a more detailed analysis, you can use Windows Performance Recorder (WPR) to capture performance data:

1. Start recording:
```
wpr -start generalprofile -filemode
```

2. Reproduce the issue or wait for the next CPU spike.

3. Stop recording:
```
wpr -stop output.etl
```
4. Analyze the output.etl file using Windows Performance Analyzer (WPA), which you can run on a machine with a GUI.

NARRAVULA MUNISAI TEJA · Answer

Hello Saini,

Here I am providing one by one steps to solve your issue.

**1. Identify the culprit pod:**

Run the following command to see which pods are currently using the most CPU:

```
kubectl top pods

```
This will display a table showing the resource usage of your pods, including CPU. Look for pods with high CPU utilization.

**2. Monitor CPU usage over time:**

To confirm the periodicity of the spikes, run this command to see the CPU utilization history of your pods:

```
kubectl top pods --history
```

This will show you how CPU usage has changed over time for your pods. Look for spikes that occur every six hours.

**3. Investigate pod logs:**

Once you've identified a suspicious pod, use this command to view its logs:

```
kubectl logs 
```

The logs might reveal clues about what's causing the CPU spikes. Look for errors, warnings, or any messages indicating high resource usage.

**Troubleshooting High CPU Usage on Kubernetes Pods: This guide provides a more in-depth approach to diagnosing CPU spikes in Kubernetes pods, though it may not be specific to Windows:** https://github.com/kubernetes/kubernetes/issues/69083

Sandeep · Answer

Hello

check these steps to resolve the issue:

Gather Resource Utilization Data:

1. 
```
kubectl top nodes: 
```
Use this command to get an overview of CPU and memory usage across all nodes in your Kubernetes cluster. Look for spikes in CPU utilization that coincide with the six-hour intervals.

```
kubectl top pods: 
```
This command displays CPU and memory usage for each pod in the cluster. Filter by namespace if needed (e.g., kubectl top pods -n my-namespace). Identify pods with high CPU usage during the spike times.
2. Identify Pod Details:

```
kubectl describe pod : Replace 
```
 with the name of a pod showing high CPU usage. This command provides detailed information about the pod, including its container images, resource requests/limits, and events.
3. Analyze Container Processes:

Connect to Pod: 
```
Use kubectl exec -it 
```
 bash to establish a bash shell session within the pod. This allows you to examine running processes.
Process Monitoring: Use top or ps auxf to view a list of running processes and their CPU utilization. Identify processes consuming significant CPU resources during the spike intervals.
Additional Tools: Tools like strace or perf (if available on your Windows Core instance) can provide deeper insights into system calls and function calls made by the CPU-intensive process, helping pinpoint the root cause. However, these require familiarity with system profiling techniques.

High CPU Utilization on Windows Core Instance with Kubernetes Pods Every Six Hours

Relevant content