gRPC streaming through ALB using ECS

I have a gRPC server running on an ECS cluster, and I've put an ALB in front of it (using protocolVersion gRPC). When trying to connect to the server, I'm able to run a method that uses a unary connection. However, once I try a bidirectional streaming endpoint, the server blocks on trying to receive the stream coming from the client. After the timeout it ends with a "Broken pipe" error. This applications runs without any problems on a docker locally, so it seems that I'm missing some network configuration. All components are in the same VPC, I have two security groups, one for the ALB allowing 443 traffic from everywhere and all outgoing traffic, the second security group is for the ECS cluster and allows all traffic from the ALB, and all outgoing traffic. I have tried to allow all traffic from the ECS cluster to the VPC as well, without any success.

Does anyone have any idea why there is this difference between unary and bidirectional streaming and how to solve this?