Does RDS Custom for SQL Server support dedicated hosts?

No. You will have to choose the instance classes shown in DB instance class support for RDS Custom for SQL Server.

RDS custom SQL Server is a license included model same as RDS SQL Server and do not support Bring Your Own License (BYOL). In RDS Custom FAQ Section "Licensing and Version Support" shares the same information of Licensing model for RDS Custom SQL Server.