What rules should I add to my custom security group to avoid using the default VPC security group and still pass the health check?

You will need to add rules to allow inbound traffic on the port that the Application Load Balancer (ALB) is using for health checks. By default this is port TCP/443 or TCP/80 depending on whether you specify HTTPS or HTTP health checks. The security group should allow ingress from the ALB security group on the health check port.

When using the bridge network mode, the containers are placed on the same network as the EC2 host instance. This means the security groups of the EC2 instance apply to the containers as well. Since the default VPC security group is associated with EC2 instances by default, it allows the health checks from the ALB to reach the containers without needing to be explicitly added.

In the awsvpc network mode, each container gets its own elastic network interface and thus its own security group.

As Giovanni mentioned, the security group needs to allow ingress from the ALB security group on the health check port.

See if following the documentation guidelines on security group configurations solves your problem here: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html

Or allowing the CIDR ranges on the target group IPv4 Addresses: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html