After upgrade to 12.3.1 - 500 errors - Loading some projects, and loading issues

Summary

Doing the following produces HTTP 500 errors:

  • /merge_requests (all projects)
  • /issues/(issue number) (All projects)
  • Some projects do not load as well

Steps to reproduce

Go to any of the above pages

Example Project

On premise gitlab-ce

What is the current bug behavior?

Pages load with 500 error

What is the expected correct behavior?

Pages load withotu error

Relevant logs and/or screenshots

image

Started GET "/GroupName/ProjectName/issues/466" for 10.211.33.102 at 2019-09-29 17:33:23 -0500
Processing by Projects::IssuesController#show as HTML
  Parameters: {"namespace_id"=>"GroupName", "project_id"=>"ProjetName", "id"=>"466"}
Completed 500 Internal Server Error in 115ms (ActiveRecord: 15.5ms | Elasticsearch: 0.0ms)

ActionView::Template::Error (undefined local variable or method `duplicated_to_id' for #<Issue id:4908 GroupName/ProjectName#466>
Did you mean?  duplicated_to
               duplicated_to=):
    15:     .issuable-status-box.status-box.status-box-issue-closed{ class: issue_status_visibility(@issue, status_box: :closed) }
    16:       = sprite_icon('mobile-issue-close', size: 16, css_class: 'd-block d-sm-none')
    17:       .d-none.d-sm-block
    18:         = issue_closed_text(@issue, current_user)
    19:     .issuable-status-box.status-box.status-box-open{ class: issue_status_visibility(@issue, status_box: :open) }
    20:       = sprite_icon('issue-open-m', size: 16, css_class: 'd-block d-sm-none')
    21:       %span.d-none.d-sm-block Open

app/models/issue.rb:186:in `duplicated?'
app/helpers/issues_helper.rb:165:in `issue_closed_link'
app/helpers/issues_helper.rb:171:in `issue_closed_text'
app/views/projects/issues/show.html.haml:18:in `_app_views_projects_issues_show_html_haml___2021487532546116329_70043871254660'
app/controllers/application_controller.rb:118:in `render'
app/controllers/concerns/issuable_actions.rb:33:in `block (2 levels) in show'
app/controllers/concerns/issuable_actions.rb:30:in `show'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Started GET "/GroupName/ProjectName" for IPAddress at 2019-09-29 17:37:07 -0500
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"GroupName", "id"=>"ProjectName"}
Completed 500 Internal Server Error in 354ms (ActiveRecord: 11.6ms | Elasticsearch: 0.0ms)

ActionView::Template::Error (undefined local variable or method `static_objects_external_storage_url' for #<ApplicationSetting:0x00007f68bea8e228>
Did you mean?  static_objects_external_storage_enabled?
               static_objects_external_storage_auth_token):
    3: .btn-group.ml-0.w-100
    4:   - formats.each do |(fmt, extra_class)|
    5:     - archive_path = project_archive_path(project, id: tree_join(ref, archive_prefix), path: path, format: fmt)
    6:     = link_to fmt, external_storage_url_or_path(archive_path), rel: 'nofollow', download: '', class: "btn btn-xs #{extra_class}"

app/models/application_setting_implementation.rb:310:in `static_objects_external_storage_enabled?'
lib/gitlab/current_settings.rb:20:in `method_missing'
app/helpers/application_helper.rb:173:in `static_objects_external_storage_enabled?'
app/helpers/application_helper.rb:177:in `external_storage_url_or_path'
app/views/projects/buttons/_download_links.html.haml:6:in `block in _app_views_projects_buttons__download_links_html_haml__4038446149004563468_70043865565640'
app/views/projects/buttons/_download_links.html.haml:4:in `each'
app/views/projects/buttons/_download_links.html.haml:4:in `_app_views_projects_buttons__download_links_html_haml__4038446149004563468_70043865565640'
app/views/projects/buttons/_download.html.haml:14:in `_app_views_projects_buttons__download_html_haml___4516140207079799051_70043864047500'
app/views/projects/tree/_tree_header.html.haml:93:in `_app_views_projects_tree__tree_header_html_haml___1499212104517624844_70043862317760'
app/views/projects/_files.html.haml:10:in `_app_views_projects__files_html_haml___2423279089574754119_70043860409360'
app/views/projects/show.html.haml:33:in `_app_views_projects_show_html_haml___2260347193900459079_70043913178040'
app/controllers/application_controller.rb:118:in `render'
app/controllers/application_controller.rb:533:in `block in allow_gitaly_ref_name_caching'
lib/gitlab/gitaly_client.rb:300:in `allow_ref_name_caching'
app/controllers/application_controller.rb:532:in `allow_gitaly_ref_name_caching'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Output of checks

?

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.22.0
Sidekiq Version:5.2.7
Go Version:     unknown


GitLab information
Version:        12.3.1
Revision:       3440d0f6100
Directory:      /var/optlinked/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.9
URL:            https://gitlab.ochsner.org
HTTP Clone URL: https://gitlab.ochsner.org/some-group/some-project.git
SSH Clone URL:  git@gitlab.ochsner.org:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:


GitLab Shell
Version:        10.0.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... no
Try fixing it:
sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
Please fix the error above and rerun the checks.
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
1/1 ... yes
4/2 ... yes
2/3 ... yes
2/4 ... yes
13/6 ... yes
4/7 ... yes
5/8 ... yes
4/9 ... yes
2/11 ... yes
4/12 ... yes
4/13 ... yes
2/14 ... yes
2/15 ... yes
2/16 ... yes
4/20 ... yes
4/21 ... yes
22/22 ... yes
9/23 ... yes
9/24 ... yes
29/25 ... yes
4/26 ... yes
2/27 ... yes
10/30 ... yes
9/31 ... yes
10/32 ... yes
9/33 ... yes
4/34 ... yes
4/35 ... yes
4/36 ... yes
2/37 ... yes
4/38 ... yes
2/39 ... yes
2/40 ... yes
4/41 ... yes
9/42 ... yes
9/44 ... yes
16/45 ... yes
9/46 ... yes
9/47 ... yes
9/48 ... yes
4/49 ... yes
2/50 ... yes
4/51 ... yes
4/52 ... yes
4/53 ... yes
4/54 ... yes
29/55 ... yes
4/56 ... yes
21/57 ... yes
21/58 ... yes
21/59 ... yes
21/60 ... yes
22/61 ... yes
4/63 ... yes
13/64 ... yes
4/65 ... yes
2/66 ... yes
2/67 ... yes
4/68 ... yes
26/69 ... yes
27/70 ... yes
9/71 ... yes
27/72 ... yes
27/73 ... yes
27/74 ... yes
27/75 ... yes
27/76 ... yes
9/77 ... yes
30/78 ... yes
30/79 ... yes
30/80 ... yes
26/81 ... yes
29/82 ... yes
29/83 ... yes
31/84 ... yes
32/85 ... yes
33/86 ... yes
9/88 ... yes
13/89 ... yes
27/90 ... yes
13/91 ... yes
26/92 ... yes
9/93 ... yes
36/94 ... yes
37/95 ... yes
26/98 ... yes
30/99 ... yes
29/100 ... yes
22/101 ... yes
2/102 ... yes
27/103 ... yes
27/104 ... yes
9/105 ... yes
43/106 ... yes
27/107 ... yes
27/108 ... yes
27/109 ... yes
27/110 ... yes
27/111 ... yes
2/112 ... yes
29/113 ... yes
29/114 ... yes
26/115 ... yes
27/116 ... yes
27/117 ... yes
27/118 ... yes
27/120 ... yes
27/121 ... yes
27/122 ... yes
27/123 ... yes
27/124 ... yes
2/125 ... yes
27/126 ... yes
27/127 ... yes
27/128 ... yes
4/134 ... yes
27/135 ... yes
27/136 ... yes
22/137 ... yes
4/138 ... yes
50/139 ... yes
27/140 ... yes
27/141 ... yes
4/142 ... yes
27/143 ... yes
27/144 ... yes
26/145 ... yes
26/146 ... yes
52/147 ... yes
27/148 ... yes
27/149 ... yes
27/150 ... yes
2/152 ... yes
2/153 ... yes
27/154 ... yes
61/156 ... yes
29/157 ... yes
27/158 ... yes
4/159 ... yes
22/160 ... yes
27/161 ... yes
26/162 ... yes
29/163 ... yes
4/164 ... yes
27/165 ... yes
27/166 ... yes
27/167 ... yes
27/168 ... yes
27/169 ... yes
4/170 ... yes
27/172 ... yes
27/173 ... yes
27/174 ... yes
4/175 ... yes
27/176 ... yes
13/177 ... yes
27/178 ... yes
26/179 ... yes
27/180 ... yes
4/181 ... yes
69/182 ... yes
69/183 ... yes
27/184 ... yes
27/185 ... yes
27/186 ... yes
13/187 ... yes
4/188 ... yes
27/189 ... yes
73/191 ... yes
27/192 ... yes
73/193 ... yes
26/194 ... yes
26/195 ... yes
29/196 ... yes
69/197 ... yes
27/198 ... yes
27/199 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 45
Is authorized keys file accessible? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

?

Edited by Drew Miranda