Skip to content

Added toggle for workhorse keywatcher

Natan Hoppe requested to merge nhoppe1/issue12812/workhorse_keywatcher_toggle into master

What does this MR do?

Adds a toggle to enable or disable [redis] block in the workhorse-config.toml file.

Why?

Workhorse has a "keywatcher" component that subscribes to a redis key in order to get notifications on certain changes. It uses this to serve long-polling requests.

This mechanism is putting a lot of load onto redis (5% of user cycles on the redis-persistent primary).

We discovered that all of our workhorse processes are subscribed to the redis key, and thus we are delivering messages to many workhorses that may not need them.

The only workhorse fleet that needs these messages is api. The proposal is to remove keywatcher from all other workhorse fleets: git, web, websockets.

What this means for self-managed GitLab

The default behavior of the webservice pod will not change and the option to disable the [redis] block was added to lower the load on Redis. The new setting is only relevant if experiencing a high rate of published messages to Redis as documented here and the workhorse.keywatcher should be disabled (set to false) for git, web, and websockets containers while it should be enabled (set to true by default) for api container.

Related issues

Infrastructure#12812 - Only enable workhorse keywatcher on API

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • Merge Request Title and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for omnibus-gitlab opened
Edited by John Skarbek

Merge request reports