AWS EC2 T2 Medium Performance Issue

Based on the information provided, it seems you're experiencing a complex performance issue with your AWS EC2 T2 Medium instance. While the CPU credits and utilization appear normal, there are a few potential areas to investigate for the root cause analysis (RCA) of the high system load occurring every 2-3 hours:

1. Application-specific issues: The high system load might be caused by periodic tasks or processes running on the instance. Review your application logs and scheduled tasks to identify any resource-intensive operations that occur at regular intervals.

2. Memory constraints: Although you mentioned RAM looks normal, T2 Medium instances have limited memory (4 GB). Ensure that your applications are not hitting memory limits, which could cause increased swap usage and higher system load.

3. I/O bottlenecks: Even though disk I/O appears normal in your monitoring tools, there might be short bursts of I/O activity causing temporary spikes in system load. Consider using more detailed I/O monitoring tools or analyzing system logs for I/O-related issues.

4. Network-related issues: Investigate if there are any network-intensive operations or external services that your instance interacts with every 2-3 hours, which could contribute to the increased system load.

5. Kernel or OS-level issues: Check for any recurring system processes, log rotations, or maintenance tasks that might be causing the periodic load increases.

6. Instance type limitations: T2 instances are burstable performance instances. Although you mentioned CPU credits are normal, consider if the workload might be hitting other limits of the T2 Medium instance type.

To further investigate:

1. Set up more granular monitoring using the CloudWatch agent to collect detailed system-level metrics.

2. Analyze system logs and application logs around the times when high system load occurs.

3. Use Linux performance analysis tools like 'sysstat' to gather more comprehensive system statistics.

4. Consider temporarily upgrading to a larger instance type to rule out resource constraints.

5. If possible, work with the customer to gain AWS console access or have them open an AWS support case for more in-depth analysis.

Without direct access to the AWS console and more detailed metrics, pinpointing the exact cause may be challenging. However, these steps should help narrow down the potential causes and provide a path for further investigation.

Sources
Amazon EC2 T2 Instances – Amazon Web Services (AWS)
Performance - AWS Support
Problem with ec2 | AWS re:Post