Sidekiq Pod takes a long time to complete Init
The sidekiq pod needs to have the ability to scale up and down quickly (opinion). There are many ways for which we can configure the sidekiq definition for a GitLab install. One of the more interesting concepts is the ability to deploy a single sidekiq deployment PER QUEUE. Without getting into the complexity this creates, let's discuss the problems with the way the current sidekiq Pod setup prevents this from being successful and also impacts overall scale of sidekiq, regardless of deployment methods. The Pod has a
dependencies init container. This container performs a check related to the database schema. This procedure takes a really long time and creates contention around scaling quickly. Due to the failure to scale quickly, this leaves things in the queue to be delayed. This script takes roughly 50 seconds on an environment that is barely utilized. I ponder if this container should be removed from the sidekiq Pod and is instead ran as part of the migrations or upgrade Pod that runs.
With GitLab.com, we can always safely assume the database will be at the appropriate schema version as this is a step in our deployment model that occurs prior to any services being updated. Due to this, this init container does not provide value, and only slows down the ability to quickly scale sidekiq. Does this container provide value for other helm installations? Probably yes, but I don't believe it should be run prior to sidekiq starting.
We need this script to run prior to an upgrade of GitLab, is there a better place for it than the sidekiq Pod, perhaps the
upgrade_check_hook of helm?
Sidekiq should start quickly, less than 50 seconds. Enough time for the configurations to be built and the sidekiq process started. Checking database schemas seems like something of a dependency that would prevent ANY pod from starting on a GitLab installation.
- Cloud: GKE
- Kubernetes: (
- Helm: (
Timestamps below supplemented from our logging mechanism.
% k logs gitlab-sidekiq-export-966444c8-gb778 dependencies # Dec 18, 2019 @ 13:43:24.016 + /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config Begin parsing .erb files from /var/opt/gitlab/templates Writing /srv/gitlab/config/resque.yml Writing /srv/gitlab/config/sidekiq_queues.yml Writing /srv/gitlab/config/gitlab.yml Writing /srv/gitlab/config/database.yml Copying other config files found in /var/opt/gitlab/templates Copying smtp_settings.rb into /srv/gitlab/config + exec /scripts/wait-for-deps Checking database connection and schema version WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell. Database Schema - current: 20191122135327, codebase: 20191122135327 # Dec 18, 2019 @ 13:44:11.048