Issue with increasing Gitaly Timeout Period using PUMA worker
Summary
We are running GitLab13.8.4-ee (f65d7b6f) in a docker container. License ID: 131382.
Some of our repo are quite heavy, which causing underlying sidekiq job termination, on submitting merge requests
with description Gitlab::Git::CommandTimedOut: 4:Deadline Exceeded.
Historically, we have injected unicorn timeouts settings via docker-compose like this unicorn['worker_timeout'] = 60. Now we have that string unicorn['worker_timeout'] = 120, just for compatibility and to avoid the situation, where unicorn timeout is still actually used.
Puma worker do not have this configuration feature of timeout set, so we're used workaround recommended here https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout and increased it to 90 sec in gitlab.rb configuration
gitlab_rails['env'] = {
'GITLAB_RAILS_RACK_TIMEOUT' => 900,
}
Default gitaly timeout cannot be set more than 57sec actually
We are receiving such error :
The form contains the following error:
Gitaly timeout default must be less than or equal to 57
Then we're tried also to increase this parameter in the .rb file, gitlab_rails['max_request_duration_seconds'] = 85
It also doesn't help. Relevant support ticked that we're found is #198469 (closed), but it's related to the Unicorn worker timeouts, and we have PUMA worker
Steps to reproduce
- Increase PUMA timeouts in gitlab.rb configuration from defaults
gitlab_rails['env'] = {
'GITLAB_RAILS_RACK_TIMEOUT' => 900,
}
- Try to increase Gitaly Timeout Period to the value more than 57.
Example Project
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
RuntimeError
------------
The maximum request duration needs to be smaller than the worker timeout (60s)
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb:257:in `parse_maximum_request_duration'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb:41:in `parse_variables'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:187:in `block in generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:185:in `each'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:185:in `generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:24:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'
Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb:
250: end
251:
252: def parse_maximum_request_duration
253: Gitlab['gitlab_rails']['max_request_duration_seconds'] ||= (worker_timeout * 0.95).ceil
254:
255: return if Gitlab['gitlab_rails']['max_request_duration_seconds'] < worker_timeout
256:
257>> raise "The maximum request duration needs to be smaller than the worker timeout (#{worker_timeout}s)"
258: end
259:
260: def public_path
261: "#{Gitlab['node']['package']['install-dir']}/embedded/service/gitlab-rails/public"
262: end
263:
264: def worker_timeout
265: service = WebServerHelper.service_name
266: user_config = Gitlab[service]
System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=16.04
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
Running handlers complete
Chef Infra Client failed. 0 resources updated in 04 seconds
Thank you for using GitLab Docker Image!
Current version: gitlab-ee=13.8.4-ee.0
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 5.0.9 Git Version: 2.29.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.8.4-ee Revision: f65d7b6f349 DB Adapter: PostgreSQL DB Version: 12.5 Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.15.1 (For installations with omnibus-gitlab package, run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)