gitlab webhook timeout setting not works after upgrade from 13.x to 14.8.3ee in self-hosted gitlab server

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

we have a self-hosted gitlab server in local LAN and enabled webhook service for a repo, we have a Python based HTTP web service to receive this webhook event and run different shell script/job for pipeline/tag/merge request event and action.

after gitlab push webhook event to our python HTTP service, python service will run different shell script/job for different event and action.

there is a shell script, parse merge_request event, after received merged action, script need to do version increase and push new tag to server, total cost time about 9-12 seconds.

but the default webhook timeout value is 10 seconds, so we had increased this gitlab webhook timeout value to 20 seconds via this document:

https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#webhook-fails-or-multiple-webhook-requests-are-triggered

edit server config file: /etc/gitlab/gitlab.rb, and set value to :

gitlab_rails['webhook_timeout'] = 20

it works very well for some months with gitlab 13.x server and no any timeout issue exist before.

Steps to reproduce

These days, we just upgraded our local gitlab server from 13.x to 14.8.3ee version, and found some webhook script can't works any more, the timeout value works with default value 10 seconds, but the config file already exist and we didn't change/update the timeou value.

we also tried to restart service/reload config file, still not work . the error webhook job Elapsed time is 10.05/10.06 and return with HTTP 500 ERR.

so we try to catch the exception with python try/except to get the error msg reason. after catch the exception, script will return HTTP 200, webhook job Elapsed time still 10.05/10.06 , but got the exception is : timeout after 10 seconds:

[INFO] HTTP Token check pass, script start
[INFO] project_id match [sw/buildroot] repo
[INFO] Merge Request Hook matched
[ERROR] Exception matched: 
Command '/data/gitlab-ci/bin/gitlab/dev_version.sh  '[wycwyhwyq]: MR#353: add demo and doc'  cdbe4cce43b578fbaaece2b7a40eda0fdbedd3fc' timed out after 10 seconds

and we confirmed again in server config file, the value is :

gitlab_rails['webhook_timeout'] = 20

Example Project

What is the current bug behavior?

edit server config file: /etc/gitlab/gitlab.rb, and set value to :

gitlab_rails['webhook_timeout'] = 20

this setting not work

What is the expected correct behavior?

it should works per document

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes