Is transactional replication supported in RDS SQL Server or RDS Custom for SQL Server?

As of Oct 2022, this is supported only on RDS Custom for SQL Server as a Publisher. RDS SQL Server can be a Subscriber. AWS regularly adds new capabilities, so be sure to check the latest updates and documentation. Here is a blog post on this topic:

Configure SQL Server replication between Amazon RDS Custom for SQL Server and Amazon RDS for SQL Server https://aws.amazon.com/blogs/database/configure-sql-server-replication-between-amazon-rds-custom-for-sql-server-and-amazon-rds-for-sql-server/