How to have different config for each NLB target groups using Kubernetes AWS Load Balancer Controller Service annotations

So I have one service (ingress-nginx) in my EKS cluster with ALB annotations to create a NLB load balancer with this config:

 service.beta.kubernetes.io/aws-load-balancer-type: external
 service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
 service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
 service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

That service expose 5 different ports using different protocols (HTTP, TCP and UDP), so it also correctly creates 5 target groups. My problem is that some of these service, I need different configuration (for example, remove Proxy Protocol v2 on the UDP target, and override the health check port).

It does not seem possible to have different configuration for each using these annotations, only globally. This wouldn't be a problem since I can update them manually via the AWS Console webpage, and it worked. My issue is that about 3-4 hours later, the ALB controller decided to "refresh" the configuration and reset everything I changed manually, and it broke my services:

{"level":"info","ts":"2023-08-30T04:27:34Z","logger":"controllers.service","msg":"successfully deployed model","service":{"namespace":"kube-system","name":"ingress-nginx-controller"}}

So is there a way to either configure these target groups individually, or to tell ALB controller NOT to auto-refresh the configuration periodically from the annotations?