Are security groups enforced when using ssm start-session with port forwarding

The security groups are not bypassed, however, the SSM agent on the instance initiates the the connection to the SSM service so the outbound rules of the security group on the instance are the ones in play. Most likely, the outbound is wide open. Minimally, the outbound rule needs to allow outbound 443 to the SSM endpoints. See: Systems Manager prerequisites.

(Recommended) Create a VPC endpoint in Amazon Virtual Private Cloud (Amazon VPC) to use with Systems Manager.
If you don't use a VPC endpoint, configure your managed instances to allow HTTPS (port 443) outbound traffic to the Systems Manager endpoints. For information, see (Optional) Create a VPC endpoint.