QuickSight connectivity from on prem Database server

Hello.

Yes, you will need to configure Site to Site VPN or DirectConnect to connect to your on-premises database.
Also, in order for QuickSight to access the on-premises database via VPN, it must be connected to the VPC.
https://docs.aws.amazon.com/quicksight/latest/user/working-with-aws-vpc.html

Alternatively, you can allow connections from the public IP used by QuickSight on the on-premises side and access the database directly.
However, unlike a VPN, this requires making the on-premises database publicly accessible, so it is not recommended for security reasons.
https://docs.aws.amazon.com/quicksight/latest/user/configure-access.html#ip-address-ranges