After restoring backup, Gitlab begins to have 500 internal server errors
Summary
After performing gitlab-backup restore BACKUP=<backup.tar> as described here , the Docker Gitlab-FOSS from 13.0.4 instance starts experiencing 500 and 504 errors. The gitlab.rb and gitlab-secrets.json files were also correctly restored from a file backup.
Steps to reproduce
These steps most likely are only for me, as a fresh install seems to work fine. It's just my backup that doesn't work. I can provide the backup if absolutely needed.
- Start Docker instance with
gitlab/gitlab-ce:13.0.4-ce.0image (or 13.0.5 or 13.0.6) - Let Gitlab install
- Import my backup from
13.0.4, then restoregitlab.rbandgitlab-secrets.jsonand performgitlab-ctl reconfigure - Restart Docker container for good measure.
- Wait 5-15 minutes while doing normal browsing through instance
- Eventually, it will stop responding to HTTP requests and either give a 500 or 504 response
Example Project
Not applicable
What is the current bug behavior?
After restoring a backup, the Gitlab Docker image gitlab/gitlab-ce:13.0.4-ce.0 (or 13.0.5 or 13.0.6) starts to 500 and 504 error within the browser.
After 5-15 minutes while doing normal activity, the instance will error out and the container will become Unhealthy.
Only way to recover is to perform a gitlab-ctl restart or a Docker container reboot
What is the expected correct behavior?
That the Docker instance does not fail after a period of time
Relevant logs and/or screenshots
Taken from Docker logs when trying to access a project page:
Completed 500 Internal Server Error in 56994ms (ActiveRecord: 33.0ms | Elasticsearch: 0.0ms | Allocations: 89257)
ActionView::Template::Error (Request takes longer than 57 seconds):
33:
34: .js-projects-list-holder{ data: { qa_selector: 'projects_list' } }
35: - if any_projects?(projects)
36: - load_pipeline_status(projects) if pipeline_status
37: - load_max_project_member_accesses(projects) # Prime cache used in shared/projects/project view rendered below
38: %ul.projects-list{ class: css_classes }
39: - projects.each_with_index do |project, i|
lib/gitlab/request_context.rb:31:in `ensure_deadline_not_exceeded!'
lib/gitlab/gitaly_client.rb:187:in `execute'
lib/gitlab/gitaly_client.rb:170:in `block in call'
lib/gitlab/gitaly_client.rb:198:in `measure_timings'
lib/gitlab/gitaly_client.rb:169:in `call'
lib/gitlab/gitaly_client/commit_service.rb:438:in `call_find_commit'
lib/gitlab/gitaly_client/commit_service.rb:292:in `find_commit'
lib/gitlab/git/commit.rb:72:in `block in find_commit'
lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'
lib/gitlab/git/commit.rb:71:in `find_commit'
lib/gitlab/git/rugged_impl/commit.rb:41:in `find_commit'
lib/gitlab/git/commit.rb:63:in `find'
app/models/repository.rb:1142:in `find_commit'
lib/gitlab/metrics/instrumentation.rb:161:in `block in find_commit'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `find_commit'
app/models/repository.rb:115:in `commit'
lib/gitlab/metrics/instrumentation.rb:161:in `block in commit'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `commit'
app/models/concerns/has_repository.rb:51:in `commit'
lib/gitlab/cache/ci/project_pipeline_status.rb:122:in `block in commit'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/gitlab/cache/ci/project_pipeline_status.rb:121:in `commit'
lib/gitlab/cache/ci/project_pipeline_status.rb:52:in `load_status'
lib/gitlab/cache/ci/project_pipeline_status.rb:23:in `block in load_in_batch_for_projects'
lib/gitlab/cache/ci/project_pipeline_status.rb:21:in `load_in_batch_for_projects'
app/helpers/projects_helper.rb:214:in `load_pipeline_status'
app/views/shared/projects/_list.html.haml:36
app/views/dashboard/projects/_projects.html.haml:1
app/views/dashboard/projects/index.html.haml:15
app/controllers/application_controller.rb:132:in `render'
app/controllers/dashboard/projects_controller.rb:60:in `block in render_projects'
lib/gitlab/gitaly_client.rb:335:in `allow_n_plus_1_calls'
app/controllers/dashboard/projects_controller.rb:59:in `render_projects'
app/controllers/dashboard/projects_controller.rb:19:in `block (2 levels) in index'
app/controllers/dashboard/projects_controller.rb:17:in `index'
app/controllers/root_controller.rb:20:in `block in index'
lib/gitlab/gitaly_client.rb:335:in `allow_n_plus_1_calls'
app/controllers/root_controller.rb:19:in `index'
app/controllers/application_controller.rb:496:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:487:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:481:in `set_locale'
lib/gitlab/error_tracking.rb:48:in `with_context'
app/controllers/application_controller.rb:546:in `sentry_context'
app/controllers/application_controller.rb:474:in `block in set_current_context'
lib/gitlab/application_context.rb:52:in `block in use'
lib/gitlab/application_context.rb:52:in `use'
lib/gitlab/application_context.rb:20:in `with_context'
app/controllers/application_controller.rb:467:in `set_current_context'
lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:125:in `call'
lib/gitlab/middleware/read_only/controller.rb:51:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'Output of checks
Happens on locally run Docker Gitlab instance
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.6.6p146
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.9
Git Version: 2.26.2
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 13.0.6
Revision: 5aa982e01ea
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.7
URL: <redacted>
HTTP Clone URL: <redacted>
SSH Clone URL: <redacted>
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.2.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/gitResults of GitLab application Check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.2.0 ? ... OK (13.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... FAIL: 4:Deadline Exceeded
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
<redacted, but all projects had output 'yes'>
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 11
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... FinishedPossible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by Evan Swinney