How can CPU utilization be low but the Average Active Sessions be high?

You are using a burstable instance. In the case of a t2.micro that means you are getting a baseline performance of 10% of a vCPU. When you use less than baseline you accumulate CPU Credits. When you use more than baseline you use CPU Credits. When you exhaust your CPU Credits you are stuck at baseline performance. So what you will see is CPU Utilization of 10%, but that means you are using 100% of baseline. You could verify you are running out of CPU Credits by looking at CPUCreditBalance and CPUCreditUsage https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.html

For more explanation on burstable instances see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html

You've addressed this in one of the two ways I'd recommend, going to a t2.small; now your baseline is 20% of a vCPU. The other way would be to use a t3.micro, which RDS supports in Unlimited Mode. Those still burst, but can accommodate much longer periods of bursting before (instead of limiting you to baseline) you are charged for the excess usage. The T3 is 30% faster than the T2, so between a faster CPU meaning you can do more without eating into your credits, the way Unlimited lets you borrow against future credits, and that if you do use up the credits you just pay more instead of being locked into the low baseline performance, the t3.micro might also solve your problem.