so I have a following construct in my OpenShift Cluster right now:
I have an Angular Application (running on 1 Pod), a Spring Boot RestAPI (running on 1 Pod) and PowerShell HTTP Listeners, on which EXO V2 modules run on (to do different things on Exchange Online, running on 10 different Pods). The issue with this EXO V2 Module is, that you have to first initiate a connection towards Exchange-Online using the Connect-ExchangeOnline CmdLet, after which you can query e. g. the mailboxes of the connected tenant using the Get-Mailboxes CmdLet.
Now, in our application, we manage 23 different Tenants currently and we have an issue with the LoadBalancing and Session Stickiness of our PowerShell Pods, as the EXO V2 Module can only keep up 1 Exchange-Online Session simultaneously.
For example currently, when two different Tenants (calling them Tenant1 and Tenant2 from now on) send (approximately) simultaneously) a Get-Mailbox request from our RestAPI to our PowerShell POD, the requests may land in the same PowerShell Pod, resulting in Tenant1 having to authenticate (takes approximately 7 seconds), then the Get-Mailbox CmdLet runs. After the execution of that, Tenant2 now authenticates (takes another 7 seconds) and then the Get-Mailbox CmdLet runs. If now Tenant1 would want to run Get-Mailboxes again, he'd have to reauthenticate.
So for a shortterm solution, we want to create a session stickiness based on a TenantId, but also want to loadbalance on a RoundRobin basis simultaneously, so that we reduce the amount of simultaneous connections on 1 PowerShell Pod as low as possible.
For this reason, I created an annotation of our PowerShell Route called haproxy.router.openshift.io/cookie_name: TenantID and added into our Header a Cookie KeyValue Pair called ("Cookie", "TenantID=8123"). This didn't help, as the cookie_name annotation seems to just make the router RETURN a randomly hashed cookie with the given name in the annotation (in our case, "TenantId") which then should be used in subsequent requests. The issue with this however is, that the router always routes to the same Pod, even when you try to send requests for 2 different TenantIds, resulting in both Tenants become sticky to the same Pod.
Is there a way to achieve a session stickiness using an manually entered HTTP-Header, that the Router can use to route accordingly to the same pod everytime, but also make sure to distribute the load evenly?