Remove a System Hooks 502 error
Summary
Cannot remove system hooks. click remove button will show 502 with message Whoops, GitLab is taking too much time to respond.
in 60s, Response GitLab is not responding
from API DELETE /hooks/:id
in 60s
Steps to reproduce
Web page:
- click
Admin area
from top menus - click
System Hooks
from left menus - click
Remove
button from one of System hooks listed
Then you will see that!
API:
DELETE https://mygitlab.domain.com/api/v4/hooks/8?private_token=TOKEN
you will see the text GitLab is not responding
in 60s
Results of GitLab application Check
Expand for output related to the GitLab application check
# gitlab-rake gitlab:env:info System information System: CentOS 7.6.1810 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.21.0 Sidekiq Version:5.2.7 Go Version: go1.11.5 linux/amd64 GitLab information Version: 11.11.1 Revision: ac0d1491ff7 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.8 URL: https://example.com HTTP Clone URL: https://example.com/some-group/some-project.git SSH Clone URL: git@example.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 9.1.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 3/2 ... yes 14/6 ... yes 4/7 ... yes 4/8 ... yes 4/9 ... yes 3/10 ... yes 1/11 ... yes 3/12 ... yes 4/13 ... yes 1/14 ... yes 4/15 ... yes 1/16 ... yes 13/19 ... yes 17/20 ... yes 17/21 ... yes 17/22 ... yes 19/24 ... yes 25/25 ... yes 13/26 ... yes 13/27 ... yes 13/28 ... yes 34/29 ... yes 34/30 ... yes 34/31 ... yes 1/33 ... yes 1/35 ... yes 42/36 ... yes 47/39 ... yes 47/40 ... yes 1/42 ... yes 50/44 ... yes 49/45 ... yes 51/46 ... yes 1/47 ... yes 13/48 ... yes 53/49 ... yes 13/50 ... yes 66/51 ... yes 67/52 ... yes 51/53 ... yes 69/54 ... yes 72/55 ... yes 72/61 ... yes 27/62 ... yes 13/63 ... yes 7/64 ... yes 75/65 ... yes 13/66 ... yes 77/67 ... yes 77/68 ... yes 77/69 ... yes 79/70 ... yes 13/71 ... yes 13/72 ... yes 13/73 ... yes 13/74 ... yes 13/75 ... yes 90/76 ... yes 90/77 ... yes 90/78 ... yes 71/79 ... yes 13/80 ... yes 90/81 ... yes 92/82 ... yes 95/83 ... yes 114/84 ... yes 1/88 ... yes 56/89 ... yes 133/92 ... yes 90/94 ... yes 110/97 ... yes 111/98 ... yes 112/99 ... yes 112/100 ... yes 114/101 ... yes 117/105 ... yes 1/106 ... yes 114/107 ... yes 114/108 ... yes 114/111 ... yes 119/112 ... yes 114/114 ... yes 120/115 ... yes 120/116 ... yes 56/117 ... yes 130/119 ... yes 130/120 ... yes 117/121 ... yes 27/122 ... yes 132/123 ... yes 96/126 ... yes 130/132 ... yes 130/133 ... yes 120/134 ... yes 119/135 ... yes 79/139 ... yes 113/140 ... yes 96/141 ... yes 67/143 ... yes 114/144 ... yes 112/145 ... yes 120/146 ... yes 114/147 ... yes 56/149 ... yes 153/150 ... yes 137/157 ... yes 114/158 ... yes 130/159 ... yes 96/160 ... yes 140/161 ... yes 140/164 ... yes 130/165 ... yes 110/166 ... yes 35/167 ... yes 145/168 ... yes 147/169 ... yes 133/172 ... yes 152/173 ... yes 114/179 ... yes 154/180 ... yes 35/181 ... yes 154/182 ... yes 117/183 ... yes 13/184 ... yes 113/185 ... yes 113/186 ... yes 113/187 ... yes 113/188 ... yes 155/189 ... yes 157/190 ... yes 157/191 ... yes 157/192 ... yes 157/193 ... yes 111/194 ... yes 112/195 ... yes 112/197 ... yes 153/198 ... yes 109/199 ... yes 114/200 ... yes 155/202 ... yes 1/205 ... yes 114/207 ... yes 114/208 ... yes 159/212 ... yes 163/213 ... yes 114/214 ... yes 167/215 ... yes 48/216 ... yes 114/218 ... yes 90/219 ... yes 159/222 ... yes 13/223 ... yes 130/224 ... yes 130/225 ... yes 111/226 ... yes 133/227 ... yes 48/228 ... yes 172/229 ... yes 173/230 ... yes 173/231 ... yes 173/232 ... yes 133/234 ... yes 114/235 ... yes 178/237 ... yes 111/238 ... yes 42/240 ... yes 183/241 ... yes 111/242 ... yes 160/243 ... yes 181/244 ... yes 111/245 ... yes 190/246 ... yes 114/247 ... yes 111/248 ... yes 108/249 ... yes 23/250 ... yes 183/251 ... yes 23/252 ... yes 113/254 ... yes 191/255 ... yes 113/256 ... yes 1/257 ... yes 120/258 ... yes 183/259 ... yes 183/260 ... yes 193/261 ... yes 1/262 ... yes 195/264 ... yes 191/265 ... yes 182/266 ... yes 193/267 ... yes 183/269 ... yes 183/270 ... yes 183/271 ... yes 11/272 ... yes 196/273 ... yes 196/274 ... yes 196/275 ... yes 133/276 ... yes 1/277 ... yes 133/279 ... yes 111/280 ... yes 201/282 ... yes 233/288 ... yes 210/289 ... yes 210/290 ... yes 210/291 ... yes 210/292 ... yes 210/293 ... yes 210/295 ... yes 210/299 ... yes 210/300 ... yes 152/302 ... yes 130/303 ... yes 210/304 ... yes 210/307 ... yes 211/308 ... yes 210/309 ... yes 210/310 ... yes 8/311 ... yes 172/312 ... yes 210/313 ... yes 210/314 ... yes 217/316 ... yes 217/317 ... yes 152/318 ... yes 210/319 ... yes 216/322 ... yes 198/323 ... yes 215/324 ... yes 214/328 ... yes 210/329 ... yes 215/330 ... yes 220/332 ... yes 209/333 ... yes 213/336 ... yes 213/337 ... yes 213/338 ... yes 198/339 ... yes 215/340 ... yes 110/341 ... yes 209/342 ... yes 1/344 ... yes 172/345 ... yes 224/346 ... yes 13/347 ... yes 203/348 ... yes 210/349 ... yes 209/350 ... yes 111/351 ... yes 111/352 ... yes 209/353 ... yes 229/354 ... yes 154/356 ... yes 209/360 ... yes 210/361 ... yes 172/362 ... yes 172/363 ... yes 220/364 ... yes 220/365 ... yes 199/366 ... yes 229/367 ... yes 130/368 ... yes 183/371 ... yes 210/372 ... yes 71/373 ... yes 41/374 ... yes 210/375 ... yes 203/376 ... yes 209/378 ... yes 247/380 ... yes 191/381 ... yes 41/382 ... yes 210/383 ... yes 163/384 ... yes 210/385 ... yes 229/386 ... yes 255/387 ... yes 209/388 ... yes 229/389 ... yes 247/390 ... yes 114/391 ... yes 209/392 ... yes 248/393 ... yes 258/394 ... yes 114/395 ... yes 298/396 ... yes 298/398 ... yes 210/399 ... yes 261/401 ... yes 209/402 ... yes 209/405 ... yes 261/409 ... yes 258/410 ... yes 172/411 ... yes 227/412 ... yes 266/414 ... yes 266/415 ... yes 271/419 ... yes 271/420 ... yes 254/421 ... yes 256/422 ... yes 170/423 ... yes 256/424 ... yes 261/425 ... yes 258/426 ... yes 183/427 ... yes 242/428 ... yes 260/429 ... yes 261/430 ... yes 1/431 ... yes 278/434 ... yes 278/435 ... yes 274/440 ... yes 275/441 ... yes 227/444 ... yes 284/445 ... yes 113/447 ... yes 275/448 ... yes 275/449 ... yes 145/450 ... yes 114/451 ... yes 30/453 ... yes 285/454 ... yes 286/455 ... yes 287/456 ... yes 289/457 ... yes 66/461 ... yes 191/462 ... yes 152/463 ... yes 267/468 ... yes 114/471 ... yes 189/472 ... yes 295/473 ... yes 269/475 ... yes 287/476 ... yes 300/477 ... yes 201/478 ... yes 303/480 ... yes 134/481 ... yes 114/482 ... yes 305/485 ... yes 305/486 ... yes 221/491 ... yes 306/494 ... yes 306/495 ... yes 189/496 ... yes 287/497 ... yes 227/498 ... yes 284/499 ... yes 228/500 ... yes 310/501 ... yes 229/502 ... yes 305/503 ... yes 200/504 ... yes 183/505 ... yes 133/506 ... yes 312/507 ... yes 198/508 ... yes 247/509 ... yes 222/511 ... yes 298/513 ... yes 271/516 ... yes 315/518 ... yes 315/519 ... yes 315/520 ... yes 298/521 ... yes 298/524 ... yes 298/525 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.5.3) Git version >= 2.21.0 ? ... yes (2.21.0) Git user has default SSH configuration? ... yes Active users: ... 160 Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Logs
# gitlab-ctl tail | grep hook
Started DELETE "/admin/hooks/8" for 192.168.1.124 at 2019-06-24 19:27:49 +0800
192.168.1.124 - - [24/Jun/2019:19:28:50 +0800] "POST /admin/hooks/8 HTTP/2.0" 502 2940 "https://example.com/admin/hooks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
2019-06-24_11:28:50.25930 time="2019-06-24T19:28:50+08:00" level=error msg=error correlation_id=RsMny9KqFDa error="badgateway: failed after 61s: EOF" method=POST uri=/admin/hooks/8
2019-06-24_11:28:50.25941 example.com 192.168.1.124:0 - - [2019/06/24:19:28:50 +0800] "POST /admin/hooks/8 HTTP/1.1" 502 2940 "https://example.com/admin/hooks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 61.064