Schema-related Sidekiq errors after a GitLab upgrade resolved by a Sidekiq restart
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
After a GitLab upgrade has been performed admins start seeing errors in Sidekiq that appear to be caused by an an out of date or corrupted DB schema.
The precise error varies, but in all cases gitlab-rake db:migrate:status shows all migrations as up and the DB console shows the correct schema for the table in question.
Restarting Sidekiq immediately resolves the issue, suggesting a caching issue.
This has occurred with Omnibus package, Omnibus Docker image, and Helm installs of GitLab.
Steps to reproduce
Issue does not occur consistency, exact steps to trigger this scenario are unknown.
What is the current bug behavior?
One or more Sidekiq queues start to consistently fail after an upgrade with errors regarding undefined . A restart of Sidekiq clears the issue.
What is the expected correct behavior?
Sidekiq works normally after an upgrade.
Relevant logs and/or screenshots
2020-08-18T15:46:46.359Z 10 TID-gnw380t7q WARN: {"context":"Job raised exception","job":{"class":"BuildQueueWorker","args":[7398],"retry":3,"queue":"pipelin
e_processing:build_queue","queue_namespace":"pipeline_processing","jid":"2bb781c23b93b970c0661daa","created_at":1597765545.2259293,"meta.user":"username","meta.project":"group/project","meta.root_namespace":"group","meta.caller_id":"NewMergeRequestWorker","correlation_id":"ae00e213c5ca3874a22803c2e74e1f43","enqueued_at":1597765606.341877,"error_message":"Unknown primary key for table ci_runners in model Ci::Runner.","error_class":"ActiveRecord::UnknownPrimaryKey","failed_at":1597765545.261891,"retry_count":1,"retried_at":1597765577.0961802,"db_count":3,"db_write_count":0,"db_cached_
count":0},"jobstr":"{\"class\":\"BuildQueueWorker\",\"args\":[7398],\"retry\":3,\"queue\":\"pipeline_processing:build_queue\",\"queue_namespace\":\"pipeline
_processing\",\"jid\":\"2bb781c23b93b970c0661daa\",\"created_at\":1597765545.2259293,\"meta.user\":\"username\",\"meta.project\":\"group/project\",\"meta.root_namespace\":\"group\",\"meta.caller_id\":\"NewMergeRequestWorker\",\"correlation_id\":\"ae00e213c5ca3874a22803c2e74e1f
43\",\"enqueued_at\":1597765606.341877,\"error_message\":\"Unknown primary key for table ci_runners in model Ci::Runner.\",\"error_class\":\"ActiveRecord::UnknownPrimaryKey\",\"failed_at\":1597765545.261891,\"retry_count\":1,\"retried_at\":1597765577.0961802}"}
2020-08-18T15:46:46.359Z 10 TID-gnw380t7q WARN: ActiveRecord::UnknownPrimaryKey: Unknown primary key for table ci_runners in model Ci::Runner.
2020-08-18T15:46:46.359Z 10 TID-gnw380t7q WARN: srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:322:in `primary_key'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:468:in `association_primary_key'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:727:in `join_primary_key'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:292:in `get_join_keys'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:755:in `get_join_keys'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:1024:in `get_join_keys'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/reflection.rb:172:in `join_keys'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:80:in `next_chain_scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:125:in `block in add_constraints'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:124:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:124:in `each_cons'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:124:in `add_constraints'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:29:in `scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association_scope.rb:7:in `scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association.rb:225:in `association_scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/association.rb:101:in `scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/collection_association.rb:289:in `scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/collection_proxy.rb:929:in `scope'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/associations/collection_proxy.rb:1101:in `reorder'
/srv/gitlab/lib/gitlab/sql/union.rb:28:in `block (2 levels) in to_sql'
/srv/gitlab/lib/gitlab/sql/union.rb:28:in `map'
/srv/gitlab/lib/gitlab/sql/union.rb:28:in `block in to_sql'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract_adapter.rb:273:in `unprepared_statement'
/srv/gitlab/lib/gitlab/sql/union.rb:27:in `to_sql'
/srv/gitlab/app/models/concerns/from_union.rb:41:in `from_union'
/srv/gitlab/app/models/project.rb:1675:in `all_runners'
/srv/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in `block in all_runners'
/srv/gitlab/lib/gitlab/metrics/method_call.rb:36:in `measure'
/srv/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in `all_runners'
/srv/gitlab/app/services/ci/update_build_queue_service.rb:6:in `execute'
/srv/gitlab/app/workers/build_queue_worker.rb:15:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/core_ext/object/try.rb:12:in `try'
/srv/gitlab/app/workers/build_queue_worker.rb:14:in `perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:192:in `execute_job'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/srv/gitlab/lib/gitlab/metrics/sidekiq_middleware.rb:16:in `block in call'
/srv/gitlab/lib/gitlab/metrics/transaction.rb:54:in `run'
/srv/gitlab/lib/gitlab/metrics/sidekiq_middleware.rb:16:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:32:in `perform'
/srv/gitlab/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:40:in `perform'
/srv/gitlab/lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'
/srv/gitlab/lib/gitlab/sidekiq_middleware/worker_context/server.rb:16:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/admin_mode/server.rb:8:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-labkit-0.12.1/lib/labkit/middleware/sidekiq/server.rb:21:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-labkit-0.12.1/lib/labkit/middleware/sidekiq/context/server.rb:14:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-labkit-0.12.1/lib/labkit/context.rb:32:in `with_context'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-labkit-0.12.1/lib/labkit/middleware/sidekiq/context/server.rb:13:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/gitlab-labkit-0.12.1/lib/labkit/middleware/sidekiq/server.rb:20:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'
/srv/gitlab/lib/gitlab/with_request_store.rb:17:in `enabling_request_store'
/srv/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/memory_killer.rb:20:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/server_metrics.rb:31:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call'
/srv/gitlab/lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'
/srv/gitlab/lib/gitlab/sidekiq_middleware/monitor.rb:7:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/marginalia-1.8.0/lib/marginalia/sidekiq_instrumentation.rb:10:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/sidekiq.rb:9:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:164:in `block in process'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/job_retry.rb:109:in `local'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/rails.rb:43:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/reloader.rb:72:in `block in wrap'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/reloader.rb:71:in `wrap'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/rails.rb:42:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:250:in `stats'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/job_logger.rb:8:in `call'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/job_retry.rb:74:in `global'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:125:in `block in dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/logging.rb:48:in `with_context'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:124:in `dispatch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:163:in `process'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:83:in `process_one'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:71:in `run'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/util.rb:16:in `watchdog'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/util.rb:25:in `block in safe_thread'2020-08-12T12:42:42.075Z {
"severity": "WARN",
"time": "2020-08-12T12:42:42.074Z",
"error_class": "ActiveModel::UnknownAttributeError",
"error_message": "unknown attribute 'locked' for Ci::Pipeline.",
"context": "Job raised exception",
"class": "PostReceive",
"args": [
"project-1",
"user-1",
"0000000000000000000000000000000000000000 REDACTED refs/heads/branch-name\n",
"{}"
],
"retry": 3,
"queue": "post_receive",
"jid": "aa0db889c399eead9750ba1e",
"created_at": "2020-08-12T12:42:41.819Z",
"meta.user": "username",
"meta.project": "group/project",
"meta.root_namespace": "group",
"meta.caller_id": "/api/:version/internal/post_receive",
"correlation_id": "n7VuwzYDYb4",
"enqueued_at": "2020-08-12T12:42:41.821Z",
"db_count": 21,
"db_write_count": 7,
"db_cached_count": 0,
"error_backtrace": [
"lib/gitlab/ci/pipeline/chain/build.rb:9:in `perform!'",
"lib/gitlab/ci/pipeline/chain/sequence.rb:20:in `block in build!'",
"lib/gitlab/ci/pipeline/chain/sequence.rb:17:in `each'",
"lib/gitlab/ci/pipeline/chain/sequence.rb:17:in `build!'",
"app/services/ci/create_pipeline_service.rb:74:in `execute'",
"app/services/ci/create_pipeline_service.rb:96:in `execute!'",
"app/services/git/base_hooks_service.rb:61:in `create_pipelines'",
"app/services/git/base_hooks_service.rb:13:in `execute'",
"app/services/git/branch_hooks_service.rb:8:in `execute'",
"app/services/git/branch_push_service.rb:71:in `execute_related_hooks'",
"app/services/git/branch_push_service.rb:28:in `execute'",
"ee/app/services/ee/git/branch_push_service.rb:13:in `execute'",
"app/services/git/process_ref_changes_service.rb:57:in `block in process_changes'",
"app/services/git/process_ref_changes_service.rb:47:in `each'",
"app/services/git/process_ref_changes_service.rb:47:in `process_changes'",
"app/services/git/process_ref_changes_service.rb:20:in `block in process_changes_by_action'",
"app/services/git/process_ref_changes_service.rb:19:in `each'",
"app/services/git/process_ref_changes_service.rb:19:in `process_changes_by_action'",
"app/services/git/process_ref_changes_service.rb:10:in `execute'",
"app/workers/post_receive.rb:108:in `process_ref_changes'",
"app/workers/post_receive.rb:57:in `process_project_changes'",
"app/workers/post_receive.rb:29:in `perform'",
"lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
"lib/gitlab/with_request_store.rb:10:in `with_request_store'",
"lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"
]
}{
"severity": "WARN",
"time": "2020-01-16T10:17:31.287Z",
"error_class": "NoMethodError",
"error_message": "undefined method `test_suites' for nil:NilClass",
"context": "Job raised exception",
"jobstr": "{\"class\":\"ReactiveCachingWorker\",\"args\":[\"MergeRequest\",62531,\"Ci::CompareTestReportsService\",null],\"retry\":3,\"queue\":\"reactive_caching\",\"jid\":\"a055e00e171cd7a149ea57cc\",\"created_at\":1579169071.0921984,\"correlation_id\":\"rkD2JR6dVK\",\"enqueued_at\":1579169851.2429094,\"error_message\":\"undefined method `test_suites' for nil:NilClass\",\"error_class\":\"NoMethodError\",\"failed_at\":1579169673.9060147,\"retry_count\":2,\"retried_at\":1579169768.9421563}",
"class": "ReactiveCachingWorker",
"args": [
"MergeRequest",
62531,
"Ci::CompareTestReportsService",
null
],
"retry": 3,
"queue": "reactive_caching",
"jid": "a055e00e171cd7a149ea57cc",
"created_at": 1579169071.0921984,
"correlation_id": "rkD2JR6dVK",
"enqueued_at": 1579169851.2429094,
"failed_at": 1579169673.9060147,
"retry_count": 2,
"retried_at": 1579169768.9421563,
"error_backtrace": [
"lib/gitlab/ci/reports/test_reports_comparer.rb:18:in `block in suite_comparers'",
"lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'",
"lib/gitlab/ci/reports/test_reports_comparer.rb:17:in `suite_comparers'",
"lib/gitlab/ci/reports/test_reports_comparer.rb:25:in `total_status'",
"app/serializers/base_serializer.rb:16:in `represent'",
"app/services/ci/compare_reports_base_service.rb:17:in `execute'",
"app/models/merge_request.rb:1310:in `calculate_reactive_cache'",
"app/models/concerns/reactive_caching.rb:136:in `block (2 levels) in exclusively_update_reactive_cache!'",
"app/models/concerns/reactive_caching.rb:183:in `enqueuing_update'",
"app/models/concerns/reactive_caching.rb:135:in `block in exclusively_update_reactive_cache!'",
"app/models/concerns/reactive_caching.rb:173:in `locking_reactive_cache'",
"app/models/concerns/reactive_caching.rb:131:in `exclusively_update_reactive_cache!'",
"app/workers/reactive_caching_worker.rb:27:in `perform'",
"lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"
]
}Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: unknown Git Version: unknown Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.2.3-ee Revision: 640e2695514 Directory: /srv/gitlab DB Adapter: PostgreSQL DB Version: 11.8 Elasticsearch: yes Geo: no Using LDAP: yes Using Omniauth: no GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/repo GitLab Shell path: /home/git/gitlab-shell Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0)
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 35 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... no
Trying to fix error automatically. ...Failed
Try fixing it:
sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input"
For more information see:
doc/install/installation.md in section "GitLab"
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
6/2 ... yes
127/7 ... yes
127/8 ... yes
8/9 ... yes
8/12 ... yes
8/59 ... yes
8/60 ... yes
6/299 ... yes
6/300 ... yes
6/388 ... yes
118/389 ... yes
86/394 ... yes
117/398 ... yes
117/399 ... yes
118/400 ... yes
192/403 ... yes
129/404 ... yes
118/405 ... yes
158/464 ... yes
158/465 ... yes
158/466 ... yes
138/467 ... yes
138/468 ... yes
158/469 ... yes
138/470 ... yes
192/471 ... yes
138/472 ... yes
138/473 ... yes
118/474 ... yes
118/475 ... yes
138/476 ... yes
138/477 ... yes
138/478 ... yes
193/479 ... yes
118/480 ... yes
118/481 ... yes
118/482 ... yes
118/483 ... yes
118/484 ... yes
158/485 ... yes
118/486 ... yes
118/487 ... yes
118/488 ... yes
158/489 ... yes
193/490 ... yes
158/491 ... yes
117/492 ... yes
118/493 ... yes
118/494 ... yes
118/495 ... yes
158/496 ... yes
117/497 ... yes
117/498 ... yes
117/499 ... yes
158/500 ... yes
118/501 ... yes
118/502 ... yes
118/503 ... yes
118/504 ... yes
117/505 ... yes
117/506 ... yes
117/507 ... yes
117/508 ... yes
117/509 ... yes
18/510 ... yes
91/511 ... yes
91/512 ... yes
118/513 ... yes
118/514 ... yes
6/517 ... yes
118/518 ... yes
179/522 ... yes
160/523 ... yes
160/524 ... yes
13/525 ... yes
9/526 ... yes
181/527 ... yes
181/528 ... yes
164/534 ... yes
171/535 ... yes
171/536 ... yes
118/537 ... yes
6/538 ... yes
122/539 ... yes
6/541 ... yes
118/542 ... yes
195/543 ... yes
181/544 ... yes
6/545 ... yes
184/546 ... yes
189/547 ... yes
21/549 ... yes
6/550 ... yes
182/551 ... yes
158/560 ... yes
158/562 ... yes
158/563 ... yes
8/564 ... yes
196/565 ... yes
196/566 ... yes
195/567 ... yes
195/569 ... yes
158/570 ... yes
169/571 ... yes
118/572 ... yes
197/573 ... yes
197/574 ... yes
158/575 ... yes
158/576 ... yes
158/577 ... yes
158/582 ... yes
158/583 ... yes
158/584 ... yes
158/585 ... yes
158/586 ... yes
158/588 ... yes
118/589 ... yes
17/590 ... yes
118/591 ... yes
158/592 ... yes
200/593 ... yes
158/608 ... yes
158/610 ... yes
6/611 ... yes
129/612 ... yes
200/613 ... yes
200/614 ... yes
189/615 ... yes
122/616 ... yes
201/618 ... yes
117/619 ... yes
158/620 ... yes
158/621 ... yes
189/622 ... yes
208/629 ... yes
158/630 ... yes
158/631 ... yes
158/632 ... yes
158/637 ... yes
158/638 ... yes
158/639 ... yes
158/640 ... yes
158/641 ... yes
158/642 ... yes
158/643 ... yes
18/644 ... yes
209/645 ... yes
209/646 ... yes
158/647 ... yes
158/649 ... yes
189/650 ... yes
213/651 ... yes
6/652 ... yes
17/653 ... yes
196/654 ... yes
196/655 ... yes
200/656 ... yes
176/657 ... yes
158/658 ... yes
160/659 ... yes
17/660 ... yes
129/661 ... yes
118/662 ... yes
218/663 ... yes
218/664 ... yes
158/665 ... yes
118/666 ... yes
217/667 ... yes
217/668 ... yes
217/669 ... yes
217/670 ... yes
217/671 ... yes
118/672 ... yes
118/673 ... yes
118/674 ... yes
218/675 ... yes
9/676 ... yes
194/677 ... yes
183/678 ... yes
160/679 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... no
Your git bin path is "/usr/bin/git"
Try fixing it:
Update your git to a version >= 2.22.0 from Unknown
Please fix the error above and rerun the checks.
Git user has default SSH configuration? ... yes
Active users: ... 35
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Elasticsearch version 6.x - 7.x? ... yes (7.7.0)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished