@takax I'll answer the questions based on my understanding, but, the product team should have more idea on this. :)
(Different domain?) If we have multiple KAS nodes, it means we have multiple IP address for them. But, rails monolith has only one IP address. So, I think it's better to have a different domain for KAS. (WDYT?)
Anything should work - either a separate domain or the GitLab domain itself. See: Agent server node settings. The IP addressed for KAS don't need to be external, they just need to be reachable to other Gitlab nodes in the cluster. Routing is done via Rails if I understand correctly.
(SSL certs for KAS domain) When we set
https://
for gitlab_kas_external_url, then it looks like Omnibus creates SSL certs for it automatically (gitlab_kas.rb#L183-188), but it didn't. (Is this correct behavior, or I'm missing something?) And then, do we need to create SSL cert files by our self?
It should, but, the protocol should be wss
(web socket secure), not https
. See: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab-kas/libraries/gitlab_kas.rb?ref_type=heads#L167 .
(Load balancer?) Is it better to set up a load balancer for multiple KAS nodes? Anyways we need a way to distribute the requests to multi nodes.
I don't think we will need a separate load balancer for KAS alone. AFAIK, the routing happens at Rails and it should distribute the load.
Also, the conf you shared seems to miss the secrets settings, see: Enable on multiple nodes. It will not work without the secrets.
Thanks a lot @clemensbeck . That fixed it. I got a new set of spec failures now. Is there a way I can run those tests locally? Looking at the config, I'll probably need to run the tests from inside a container to avoid resolving all depenendencies locally. I am also going on leave from next week for two months, so, I am short on time to do a trial-and-error to setup a local testing environment. Any help would be appreciated :) .
Priyan Sureshbabu (7ec4e990) at 26 Mar 23:53
Ah, I think it is just KAS. Web socket connections dont seem to be routed through Nginx at the moment: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/deaa5c65a16de9faab620c66e831ae21d0330bcb/files/gitlab-cookbooks/gitlab-kas/libraries/gitlab_kas.rb#L120.
I think it is just KAS and Nginx, but, note that the IP address that KAS listens on must be reachable by the Rails nodes. The steps I suggested are based on this comment. I am not sure if anything changed after that.
Priyan Sureshbabu (755a686b) at 12 Mar 14:07
test
Priyan Sureshbabu (6450ca5c) at 06 Mar 09:36
Fix variable prefix