Skip to content

Make kas prefer routing through self vs other instances, speculative routing

Mikhail Mazurskiy requested to merge ash2k/route-to-self into master

Fixes #251 (closed).

Closes gitlab-org/gitlab#320732 (closed).

Closes gitlab-org/gitlab#323131 (closed).

See individual commits:

  1. First commit is a refactoring plus kas now goes via in-memory pipe when talking to itself rather than over network. Move kas private API client pool construction to the file where the private API server is constructed. It's logical and we can also reuse e.g. jwtSecret (it was read twice before). It's also good to reduce the size of configured_app.go.
  2. Second commit makes kas always connect to self when looking for tunnels. This is done ASAP and concurrently with even the first Redis scan. If there is a tunnel, it'll be found very quickly. If not - we'll find it if it appears and it will go over in-memory pipe.
  3. Third commit changes logic to connect up to two (vs one) new kas instances on each poll. Polls are every 50ms. So quite quickly kas will be connected to all running kas instances which had a tunnel for that agent id at some point during polling. That way if an agent connects, it will be used immediately even if it connected just after a poll has been done. This (and the previous commit) is cheap thanks to !768 (merged) - if multiple tunnels are found concurrently by multiple kas instances, they are not discarded as it was before that MR, but re-registered. So speculatively connecting to more instances is not causing any external (agent-kas) traffic, only kas-kas traffic. The benefit is that tunnels are found quicker.

With the above (especially third commit), I think Redis pub-sub is not needed. In fact, it'd create a lot of extra kas-redis traffic if there was a notification for each tunnel.

Edited by Mikhail Mazurskiy

Merge request reports