Official/AWS team provided cloud formation template for AWS RDS for SQL Server + Management Instance(windows)

At this point and time there's no specific official AWS-provided CloudFormation template for the setup you described. However, you can create your own template using the AWS::RDS::DBInstance and AWS::EC2::Instance resources.

Key points to consider are:

Updating a AWS::RDS::DBInstance could replace the DB instance and lead to data loss. Always backup via snapshots before updates. On deletion, a snapshot is saved if DBClusterIdentifier is not specified, while the instance is deleted if it is. Example snippet for your scenario (update parameters as needed):

Resources:
  MyDB:
    Type: "AWS::RDS::DBInstance"
    Properties:
      AllocatedStorage: 20
      DBInstanceClass: "db.t2.small"
      Engine: "sqlserver-se"
      EngineVersion: "14.00.3049.1.v1"
      MasterUsername: "MyName"
      MasterUserPassword: "MyPassword"
      VPCSecurityGroups: ["sg-085912345678492d2"]
  MyInstance:
    Type: "AWS::EC2::Instance"
    Properties: 
      InstanceType: "t2.micro"
      ImageId: "ami-0ff8a91507f77f867"
      KeyName: "my-key-pair"

For more details, visit the AWS CloudFormation User Guide.