Application based stickiness in AWS application load balancer (target groups)

Hello,

We have multiple applications behind several ALBs. Apps are not isolated and communicate with other apps essentially making a chain of requests to reach one of the target services behind the final ALB Target Group. We need to ensure that all subsequent calls will lend onto the same instance as the first one. This approach worked pretty well for us until we switched from Classic ELB to ALB.

In the provided mockup all four Target Groups configured with Stickiness enabled (Application-based cookie) and the same cookie name "_sticky".

Each service API implementation receives the Cookie "_sticky" in the Request Header and after processing sets Set-Cookie "_sticky" in Response Header, which would have the same value. A service may place a call no another service API behind a different ALB, in that case a new request will also have the "_sticky" Cookie in the Request Header.

The expectation was to see the client "Cookie" "_sticky" in the Client API Request Header and "Set-Cookie" "_sticky" having the same value in Response Header. Each individual service in the chain is being able to log "inbound" and "outbound" value of "_sticky" Cookie which matches the expectations.

The result of the call is different though between several iterations. On the first iteration Set-Cookie for "_sticky" was not present in the Response Header, a different one was present instead:

Set-Cookie:
AWSALBAPP-0=AAAAAAAAAAAomsT8U7daSQvmNX95mxmch0W7q+wB1fA6VsdPqjFtSYPiHA82oWQStLAYPGcsodtn8GKUeHuiEoAJLAsuT37m5eFIfMlaxO+lvDDicWIC7kM0Hs95VKrLSMGLe/tiA8rxtFd3LXk4y8nJhgxu0v4Je+u/k9uf7yQnLvSeNZZY+Dqqv1DRuuBK0OXKa7dWkuZLPUE94It0QQ==; Expires=Tue, 15 Aug 2023 20:21:31 GMT; Path=/; SameSite=None; Secure
Set-Cookie:
AWSALBAPP-1=_remove_; Expires=Tue, 15 Aug 2023 20:21:31 GMT; Path=/; SameSite=None; Secure
Set-Cookie:
AWSALBAPP-2=_remove_; Expires=Tue, 15 Aug 2023 20:21:31 GMT; Path=/; SameSite=None; Secure
Set-Cookie:
AWSALBAPP-3=_remove_; Expires=Tue, 15 Aug 2023 20:21:31 GMT; Path=/; SameSite=None; Secure

On the second iteration it would either go against the same node in TG-Raw, which is a desired outcome or against a new node, which wouldn't have the necessary info, which is highly undesirable. The situation will alternate randomly only in the second request. All the subsequent requests would follow the lead of the second.

I know the setup is quite far from an ideal, but it's a legacy approach which we can't easily change at the moment. Please advise.