Skip to content

Error 500 when opening any project after updating to 10.5.2

Summary

I have a GitLab CE environment set up on my server for small personal projects. I recently updated it from 10.4.4 to 10.5.2 and only just realized that all projects except the one I currently use the most are now inaccessible and gives a "Something went wrong" 500 Internal Server Error when attempting to access them. I have 8 projects and only 1 of them doesn't give me the Error 500 right now.

Steps to reproduce

Step 1: Upgrade from 10.4.4 to 10.5.2

Step 2: Log in to self hosted GitLab.

Step 3: Open a project.

Step 4: Something went wrong (500) is shown.

What is the current bug behavior?

All projects except for one cannot be accessed and a Error 500 is returned to the user.

What is the expected correct behavior?

The projects should have been accessible properly as it should have.

Relevant logs and/or screenshots

I had no idea what log to look at, so I referred to other issues to get an idea of what to look for and where to find it. Once I found out that production.log is what I needed to find, I opened it up and immediately found the error after going to the bottom of the log. It looks like a database error however I have no idea how to fix it myself.

production.log as follows :

Completed 200 OK in 0ms (ActiveRecord: 0.0ms)
Processing by MetricsController#index as HTML
Completed 200 OK in 2ms (Views: 0.8ms | ActiveRecord: 0.0ms)
Started GET "/YandolsZX/Icarus-Imeji" for [redacted] at 2018-03-06 01:39:06 +0800
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms)
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms)
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"YandolsZX", "id"=>"Icarus-Imeji"}
Read fragment views/YandolsZX/Icarus-Imeji/f0fac16f797d4f4527d9104ff209cb838e7acfb6/application_settings/1-20180128235014032382000/false/false/en/ae797fc72ed2d4cfe1aa7d0426430b72 (0.4ms)
Completed 500 Internal Server Error in 430ms (ActiveRecord: 37.1ms)

ActionView::Template::Error (PG::UndefinedTable: ERROR:  relation "user_callouts" does not exist
LINE 5:                WHERE a.attrelid = '"user_callouts"'::regclas...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"user_callouts"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
):
    184:                     Environments
    185: 
    186:             - if project_nav_tab? :clusters
    187:               - show_cluster_hint = show_gke_cluster_integration_callout?(@project)
    188:               = nav_link(controller: [:clusters, :user, :gcp]) do
    189:                 = link_to project_clusters_path(@project), title: _('Kubernetes'), class: 'shortcuts-cluster' do
    190:                   %span
  app/helpers/user_callouts_helper.rb:12:in `user_dismissed?'
  app/helpers/user_callouts_helper.rb:6:in `show_gke_cluster_integration_callout?'
  app/views/layouts/nav/sidebar/_project.html.haml:187:in `block in _app_views_layouts_nav_sidebar__project_html_haml___3288687455951704904_69928563667000'
  app/helpers/tab_helper.rb:55:in `nav_link'
  app/views/layouts/nav/sidebar/_project.html.haml:149:in `_app_views_layouts_nav_sidebar__project_html_haml___3288687455951704904_69928563667000'
  app/views/layouts/_page.html.haml:3:in `_app_views_layouts__page_html_haml___1258951775622053518_69928416840680'
  app/views/layouts/application.html.haml:8:in `_app_views_layouts_application_html_haml___910304661803239554_69928424122700'
  app/views/layouts/project.html.haml:14:in `_app_views_layouts_project_html_haml___1391579587303322604_69928471288340'
  lib/gitlab/i18n.rb:47:in `with_locale'
  lib/gitlab/i18n.rb:53:in `with_user_locale'
  app/controllers/application_controller.rb:325:in `set_locale'
  lib/gitlab/middleware/multipart.rb:95:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/read_only.rb:31:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'

Output of checks

Ran sudo gitlab-rake gitlab:check SANITIZE=true, sudo gitlab-rake gitlab:git:fsck and sudo gitlab-rake gitlab:uploads:check and none of which reported any problems.

Results of GitLab environment info

Expand for output related to GitLab environment info

System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown

GitLab information Version: 10.5.2 Revision: b951e0d Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: [redacted] HTTP Clone URL: [redacted]/some-group/some-project.git SSH Clone URL: [redacted]:some-group/some-project.git Using LDAP: no Using Omniauth: no

GitLab Shell Version: 6.0.3 Repository storage paths:

  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check
Checking GitLab Shell ...

GitLab Shell version >= 6.0.3 ? ... OK (6.0.3) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/1 ... repository is empty 2/2 ... ok 2/3 ... repository is empty 2/4 ... ok 3/5 ... repository is empty 2/6 ... ok 2/7 ... repository is empty 2/8 ... repository is empty Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/1 ... yes 2/2 ... yes 2/3 ... yes 2/4 ... yes 3/5 ... yes 2/6 ... yes 2/7 ... yes 2/8 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 3

Checking GitLab ... Finished