Blue/Green on Fargate Load Balancer swap Target Group after deploy

Hi all,
we are setting up a service to AWS Fargate, we want to use a Blue/Green deployment.
We have an Application Load Balancer, and two Target Group, and also we have setup a CodePipeline.
The code Pipeline connects to our bitbucket repository, and build the Docker image of our service with no problem. It then add two artifacts "taskdef.json" and "appspec.yaml" that we use to deploy the app.
On the Load Balancer we have setup two Listener, on port 80 forwarded to TargetGroup1, on port 8081 forwarded to TargetGroup2.
The Deployment Group has port 8081 as Test listener port.
It all works on first time deploy, it test the new service on TargetGroup2, and deregister the service on TargetGroup1.
After the deployment the new service is registered to TargetGroup2 and BOTH the listener 80, and 8081 are forwarded to TargetGroup2.
THIS IS WHERE PROBLEMS CAMES IN.
This means the next time we deploy the "Stage 2: Test traffic route setup" will "skip" cause the status of the listener on TargetGroup1 is "Target group is not configured to receive traffic from the load balancer". This will take the Deploy directly to the next stage, that change the Load Balancer listener on BOTH port forwarding to TargetGroup1, but our service is still "initializing" and not in a "Healty" status.
I though after the deploy the testing port would be forwarded (swapped) back to the not blue group.
I'm for sure missing something, can someone please show me where I make wrong things.

Thank you!