RabbitMQ (web) behind NLB only works over IPv6, not IPv4

I have a strange issue: I have a private RabbitMQ (AWS hosted) and I want to enable the web-interface for certain IPs to allow easy access. (I don't want to use a VPN at this point). I have followed this guide: https://aws.amazon.com/blogs/compute/creating-static-custom-domain-endpoints-with-amazon-mq-for-rabbitmq/ but with one change. I don't use a custom subnet with NACL, I use "my" private subnet with a SG attached to RabbitMQ.

RabbitMQ:

Loadbalancer:

LB Listener:

LB Target Group (ignore the draining target for now):

When I do a CURL call (or open it in my browser) I only get the RabbitMQ webinterface when on IPv6, NOT on IPv4

When I deregister the RabbitMQ IP (.83) as target and register a EC2 instance (.185) as target it does work! Both IPv4 and IPv6.

In this case the EC2 instance is just a regular ubuntu, with nginx installed and a self-signed certificate in the SAME subnet as RabbitMQ

Please note: when I turn OFF "preserve client IP addresss" (and allow 0.0.0.0/0 in the SG) it does work. When I turn ON "preseve client IP address" (even with 0.0.0.0/0) it does NOT work

What am I doing wrong here? Is this not supported? Does RabbitMQ filter out non-local IPs or something?