Unable to create wiki page in certains project under group
Summary
When trying to create wiki page under certain project gitlab return 503 - Whoops, Gitlab is currently unavailable...
I know it's propably data related but I'm seeking for some advices where to look and what to try.
Gitlab version: 11.5.3 (aa11737)
Omnibus Installation under Debian Strech
Steps to reproduce
I tried it on several projects and several instances of gitlab. It's hard to reproduce maybe somehow related with our data/repositories in group or it has some relation with upgrades.
In short
- creating wiki under project, fail
- creating new project under same group, then create wiki, fail
- creating new project under another group, then wiki, fail
- creating new group, new project, wiki OK
- when transfer project from Group to me, starts Working OK
- when transfer same project from me to Group, starts failing again
Relevant logs and/or screenshots
Found following in the logs:
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2018-12-07 16:52:53 +0100
Started GET "/tech/edrive/wikis/home?view=create" for 192.168.1.249 at 2018-12-07 16:52:54 +0100
Processing by Projects::WikisController#show as HTML
Parameters: {"view"=>"create", "namespace_id"=>"tech", "project_id"=>"edrive", "id"=>"home"}
Completed 200 OK in 75ms (Views: 49.1ms | ActiveRecord: 3.6ms)
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2018-12-07 16:52:54 +0100
Started GET "/uploads/-/system/group/avatar/35/star-trek-comm-badge-tng-png-logo-18.png?width=15" for 192.168.1.249 at 2018-12-07 16:52:54 +0100
Started GET "/uploads/-/system/project/avatar/176/0b7e911056fe2a0dad1d22608761748d.png?width=15" for 192.168.1.249 at 2018-12-07 16:52:54 +0100
Started GET "/uploads/-/system/project/avatar/176/0b7e911056fe2a0dad1d22608761748d.png?width=40" for 192.168.1.249 at 2018-12-07 16:52:54 +0100
Started GET "/uploads/-/system/user/avatar/15/avatar.png?width=23" for 192.168.1.249 at 2018-12-07 16:52:54 +0100
Processing by UploadsController#show as HTML
Processing by UploadsController#show as HTML
Parameters: {"width"=>"23", "model"=>"user", "mounted_as"=>"avatar", "id"=>"15", "filename"=>"avatar.png"}
Parameters: {"width"=>"40", "model"=>"project", "mounted_as"=>"avatar", "id"=>"176", "filename"=>"0b7e911056fe2a0dad1d22608761748d.png"}
Processing by UploadsController#show as HTML
Processing by UploadsController#show as HTML
Parameters: {"width"=>"15", "model"=>"project", "mounted_as"=>"avatar", "id"=>"176", "filename"=>"0b7e911056fe2a0dad1d22608761748d.png"}
Parameters: {"width"=>"15", "model"=>"group", "mounted_as"=>"avatar", "id"=>"35", "filename"=>"star-trek-comm-badge-tng-png-logo-18.png"}
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/user/avatar/15/avatar.png (0.1ms)
Completed 200 OK in 10ms (ActiveRecord: 1.3ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/project/avatar/176/0b7e911056fe2a0dad1d22608761748d.png (0.1ms)
Completed 200 OK in 13ms (ActiveRecord: 2.3ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/group/avatar/35/star-trek-comm-badge-tng-png-logo-18.png (0.1ms)
Completed 200 OK in 16ms (ActiveRecord: 1.9ms)
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/project/avatar/176/0b7e911056fe2a0dad1d22608761748d.png (0.1ms)
Completed 200 OK in 17ms (ActiveRecord: 2.2ms)
Started POST "/tech/edrive/wikis" for 192.168.1.249 at 2018-12-07 16:52:56 +0100
Processing by Projects::WikisController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "wiki"=>{"title"=>"home", "format"=>"markdown", "content"=>"[FILTERED]", "message"=>"Create home"}, "namespace_id"=>"tech", "project_id"=>"edrive"}
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2018-12-07 16:52:56 +0100
Gitlab::Git::CommandError (2:Hook failed: ):
lib/gitlab/git/wraps_gitaly_errors.rb:11:in `rescue in wrapped_gitaly_errors'
lib/gitlab/git/wraps_gitaly_errors.rb:4:in `wrapped_gitaly_errors'
lib/gitlab/git/wiki.rb:70:in `write_page'
app/models/project_wiki.rb:115:in `create_page'
app/models/wiki_page.rb:198:in `block in create'
app/models/wiki_page.rb:313:in `save'
app/models/wiki_page.rb:197:in `create'
app/services/wiki_pages/create_service.rb:9:in `execute'
lib/gitlab/metrics/instrumentation.rb:159:in `block in execute'
lib/gitlab/metrics/method_call.rb:34:in `measure'
lib/gitlab/metrics/instrumentation.rb:159:in `execute'
app/controllers/projects/wikis_controller.rb:63:in `create'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:419:in `set_locale'
lib/gitlab/middleware/multipart.rb:101:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:17:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:22:in `call'
lib/gitlab/metrics/rack_middleware.rb:15:in `block in call'
lib/gitlab/metrics/transaction.rb:53:in `run'
lib/gitlab/metrics/rack_middleware.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:38:in `call'
lib/gitlab/middleware/read_only.rb:16:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
lib/gitlab/middleware/release_env.rb:10:in `call'
Completed 503 Service Unavailable in 130ms (Views: 6.5ms | ActiveRecord: 1.7ms)
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2018-12-07 16:52:57 +0100
Started GET "/" for 192.168.248.235 at 2018-12-07 16:52:58 +0100
Processing by RootController#index as HTML
Redirected to https://192.168.249.32/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 3ms (ActiveRecord: 0.0ms)
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2018-12-07 16:52:58 +0100
Started GET "/admin/logs" for 192.168.1.249 at 2018-12-07 16:53:00 +0100
Processing by Admin::LogsController#show as HTML
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
-> gitlab-rake gitlab:env:info Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.System information System: Debian 9.6 Current User: git Using RVM: no Ruby Version: 2.4.5p335 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.1 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.1 Go Version: unknown
GitLab information Version: 11.5.3 Revision: aa11737 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.livesport.eu HTTP Clone URL: https://gitlab.livesport.eu/some-group/some-project.git SSH Clone URL: git@gitlab.livesport.eu:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2
GitLab Shell Version: 8.4.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
No errors on gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...GitLab Shell version >= 8.4.1 ? ... OK (8.4.1) hooks directories in repos are links: ... 19/4 ... ok 19/5 ... ok 20/6 ... ok 20/7 ... ok 14/8 ... ok 19/9 ... ok 19/10 ... ok 19/11 ... ok 19/12 ... ok 19/13 ... ok 21/14 ... ok 19/15 ... ok 19/16 ... ok 14/17 ... ok 27/18 ... ok 14/19 ... ok 19/20 ... ok 34/21 ... ok 13/22 ... ok 19/23 ... ok 27/24 ... ok 37/25 ... ok 39/26 ... ok 34/27 ... ok 14/28 ... ok 67/29 ... ok 35/31 ... ok 14/32 ... ok 50/33 ... ok 67/34 ... ok 27/35 ... ok 55/37 ... ok 53/38 ... ok 13/39 ... ok 58/40 ... ok 97/41 ... ok 97/42 ... ok 35/44 ... ok 14/45 ... ok 27/46 ... ok 50/48 ... ok 31/51 ... ok 25/53 ... ok 69/54 ... ok 25/55 ... ok 70/56 ... ok 27/57 ... ok 58/58 ... ok 25/59 ... ok 27/60 ... ok 14/61 ... ok 67/62 ... ok 25/66 ... ok 14/68 ... ok 35/69 ... ok 79/70 ... ok 79/71 ... ok 25/73 ... ok 27/74 ... ok 27/75 ... repository is empty 14/76 ... ok 27/77 ... ok 27/78 ... repository is empty 83/79 ... ok 83/80 ... ok 83/81 ... ok 83/82 ... ok 64/84 ... ok 27/85 ... ok 21/86 ... ok 84/87 ... ok 86/88 ... ok 86/89 ... ok 86/90 ... ok 86/91 ... ok 86/92 ... ok 86/93 ... ok 20/94 ... ok 27/95 ... ok 87/97 ... ok 86/99 ... ok 31/100 ... ok 64/101 ... ok 27/105 ... ok 31/106 ... ok 32/107 ... ok 88/108 ... ok 68/110 ... ok 40/111 ... ok 67/112 ... ok 88/113 ... ok 25/115 ... ok 35/116 ... ok 68/118 ... ok 67/119 ... ok 90/120 ... ok 90/122 ... ok 90/123 ... ok 90/124 ... ok 27/125 ... ok 64/127 ... ok 14/128 ... ok 94/130 ... ok 95/132 ... ok 19/133 ... ok 97/135 ... ok 97/136 ... ok 97/137 ... ok 96/138 ... ok 98/139 ... ok 98/140 ... ok 97/141 ... ok 29/142 ... ok 40/143 ... repository is empty 67/144 ... ok 98/145 ... ok 97/146 ... ok 25/147 ... ok 97/148 ... ok 97/149 ... ok 14/151 ... ok 91/153 ... ok 14/154 ... ok 100/155 ... ok 100/156 ... ok 40/157 ... repository is empty 14/158 ... ok 100/159 ... ok 100/160 ... ok 19/161 ... ok 100/162 ... ok 100/163 ... ok 108/164 ... ok 14/165 ... ok 40/166 ... ok 19/167 ... ok 27/168 ... ok 101/169 ... ok 32/170 ... ok 97/171 ... ok 25/172 ... ok 67/173 ... ok 31/174 ... ok 35/175 ... ok 35/176 ... ok ... NOTHING INTERESTING ...
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 19/4 ... yes 19/5 ... yes 20/6 ... yes 20/7 ... yes 14/8 ... yes 19/9 ... yes 19/10 ... yes 19/11 ... yes 19/12 ... yes 19/13 ... yes 21/14 ... yes 19/15 ... yes 19/16 ... yes 14/17 ... yes 27/18 ... yes 14/19 ... yes 19/20 ... yes 34/21 ... yes 13/22 ... yes 19/23 ... yes ..... SAME FOR ALL.... Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.5) Git version >= 2.9.5 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 66
Checking GitLab ... Finished