does RDS Maintenance update operating system reboots the RDS instance?

Hello,

It depends on what type of maintenance action it is. There are few OS maintenance operation which doesn't requires downtime and the patch will be applied online without rebooting the instance.

Reboot will be there only when there is an offline patch applied. AWS communicates to the customer via email and PHD dashboard if there are any maintenance activity scheduled and the impact of that maintenance activity.

I have a pending O/S update in my RDS for MySQL instance but PHD doesn't show it, nor if a failover will be needed.