OpenSSL::Cipher::CipherError () - but only for specific tags used by runner
Summary
After migrating GitLab-CE from an Ubuntu 14.04 to 18.04 its impossible to use an GitLab-Runner with the tags "Behat, Composer, PHP 7.1". Every other tag-combination is fine. But whenever this 3 tags are part of the tags (or the only tags), the runner gets no jobs and in production.log are "OpenSSL::Cipher::CipherError ()" errors.
These errors do not occur on Ubuntu 14.04.
Steps to reproduce
This is the path we took to get the error:
- Setup Ubuntu 18.04 with GitLab-CE omnibus packet
- Do a backup on the Ubuntu 14.04 GitLab-CE using the GitLab backup metho
- Copying secrets.json (double checked this even with sha256 hashes, to make sure this is not the error)
- Import the backup via GitLab
- Go to the runner page, add runner with the given tags
- Have a look into production.log
I'm not sure, if this is a special case only happening to us or a general error. It is also nothing specific to the runner. We have runner on FreeBSD, Ubuntu and Windows-machines. Using the tag on any of them, creates the same error. Even complete fresh runner suffer from this.
Also it is the very specific combination. This is fine:
- Behat, PHP 7.1
- Composer, PHP 7.1
- Behat, Composer
- Behat
- Composer
- PHP 7.1
This does not work:
- Behat, Composer, PHP 7.1
- APHPUnit, Behat, Composer, PHP 7.1
- Behat, Composer, PHP 7.1, Windows
What is the current bug behavior?
Runner with this tags do not get any jobs at all. An error is logged (see later).
What is the expected correct behavior?
Runner must get a job when all tags are matched.
Relevant logs and/or screenshots
Started POST "/api/v4/jobs/request" for xxx.xxx.xxx.xxx at 2019-03-05 08:46:50 +0000
Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON
Parameters: {"info"=>{"name"=>"gitlab-runner", "version"=>"11.7.0", "revision"=>"8bb608ff", "platform"=>"freebsd", "architecture"=>"amd64", "executor"=>"shell", "shell"=>"bash", "features"=>{"variables"=>"[FILTERED]", "image"=>false, "services"=>false, "artifacts"=>true, "cache"=>true, "shared"=>true, "upload_multiple_artifacts"=>true, "upload_raw_artifacts"=>true, "session"=>true, "terminal"=>true}}, "token"=>"[FILTERED]", "last_update"=>"36e7fcccbd4764a760a79b11ff421699", "request_forgery_protection"=>{"info"=>{"name"=>"gitlab-runner", "version"=>"11.7.0", "revision"=>"8bb608ff", "platform"=>"freebsd", "architecture"=>"amd64", "executor"=>"shell", "shell"=>"bash", "features"=>{"variables"=>"[FILTERED]", "image"=>false, "services"=>false, "artifacts"=>true, "cache"=>true, "shared"=>true, "upload_multiple_artifacts"=>true, "upload_raw_artifacts"=>true, "session"=>true, "terminal"=>true}}, "token"=>"[FILTERED]", "last_update"=>"36e7fcccbd4764a760a79b11ff421699"}}
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms)
OpenSSL::Cipher::CipherError ():
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in `ensure_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:382:in `block in make_lambda'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:169:in `block (2 levels) in halting'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:770:in `block (2 levels) in deprecated_false_terminator'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:769:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:769:in `block in deprecated_false_terminator'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:170:in `block in halting'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:101:in `__run_callbacks__'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in `_run_save_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/callbacks.rb:298:in `create_or_update'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/persistence.rb:129:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/validations.rb:44:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:319:in `block (2 levels) in save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:211:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:319:in `block in save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:334:in `rollback_active_record_state!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:318:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/suppressor.rb:41:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `block in save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `block in around_save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:224:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:128:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:371:in `before'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `block in pausable'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in `pausable'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:127:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:182:in `block in within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:171:in `block in within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `block in within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:555:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:211:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:554:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in `within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in `within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `around_save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block (2 levels) in run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in `block in run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:60:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in `fire'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in `block in add_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:224:in `block in add_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:90:in `assign_runner!'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:45:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/delegation.rb:38:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/delegation.rb:38:in `each'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:39:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:117:in `block (2 levels) in <class:Runner>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:17:in `block in <class:Constraints>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:46:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:39:in `block in serve'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:727:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:103:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/correlation_id.rb:15:in `use_id'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/batch-loader-1.2.2/lib/batch_loader/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:40:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:90:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:36:in `call_app'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in `block in tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/request_id.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sentry-raven-2.7.4/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/engine.rb:522:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:13:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Results of GitLab environment info
Old machine:
Expand for output related to GitLab environment info
sudo gitlab-rake gitlab:env:info
System information System: Ubuntu 14.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: unknown
GitLab information Version: 11.7.5 Revision: c5b5b18 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab-2.anymotion.de HTTP Clone URL: https://gitlab-2.anymotion.de/some-group/some-project.git SSH Clone URL: git@gitlab-2.anymotion.de:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 8.4.4 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
New machine:
Expand for output related to GitLab environment info
$ sudo gitlab-rake gitlab:env:info [sudo] password for torstenzuehlsdorff:System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown
GitLab information Version: 11.8.1 Revision: 657d5085256 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.anymotion.de HTTP Clone URL: https://gitlab.anymotion.de/some-group/some-project.git SSH Clone URL: git@gitlab.anymotion.de:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 8.4.4 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Old machine:
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes [many yes ..] Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1551863827 sudo mv /var/opt/gitlab/.ssh/jenkins11 ~/gitlab-check-backup-1551863827 sudo mv /var/opt/gitlab/.ssh/id_dsa_git ~/gitlab-check-backup-1551863827 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 57
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
New machine:
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes [many yes ..] Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1551863617 sudo mv /var/opt/gitlab/.ssh/id_rsa_git ~/gitlab-check-backup-1551863617 sudo mv /var/opt/gitlab/.ssh/jenkins11 ~/gitlab-check-backup-1551863617 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 58
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished