Snippets did not migrate to repositories from v12.10.3 to v13.1.0
Hello,
I've launched the upgraded GitLab v13.1.0 Docker container about 3 hours ago. None of the snippets are migrating, and they all give Server Error 500 when attempting to open said snippets.
Upon checking the rails console, it reports migrations are running, but nothing is being processed.
How do I go about troubleshooting this issue?
gitlab-rails console
--------------------------------------------------------------------------------
GitLab: 13.1.0 (d62ee60b669) FOSS
GitLab Shell: 13.3.0
PostgreSQL: 11.6
--------------------------------------------------------------------------------
Loading production environment (Rails 6.0.3.1)
gitlab-rake gitlab:snippets:migration_status
There are snippet migrations running
gitlab-rake gitlab:snippets:list_non_migrated
There are 88 snippets that haven't been migrated. Showing a batch of ids of those snippets:
70,78,8,54,32,21,44,84,37,41,19,35,82,116,1,2,3,38,67,34,29,4,5,6,7,9,20,11,12,13,14,15,16,18,22,23,24,25,26,27,28,50,30,31,33,39,40,55,42,43,45,47,48,49,51,52,56,57,58,59,60,68,61,62,63,88,64,65,66,71,72,115,74,73,83,69,75,76,77,46,117,79,80,81,86,87,119,118I tried to migrate a single snippet as a test, but it refuses to run, since it says migrations are currently running:
gitlab-rake gitlab:snippets:migrate SNIPPET_IDS=1
rake aborted!
There are already snippet migrations running. Please wait until they are finished.
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/snippets.rake:19:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:snippets:migrate
(See full trace by running task with --trace)There are some errors trying to get the env info:
gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.6.6p146
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.9
Git Version: 2.27.0
Sidekiq Version:5.2.7
Go Version: unknown
rake aborted!
NoMethodError: undefined method `first' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting_implementation.rb:314:in `pick_repository_storage'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:20:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:23:in `pick_repository_storage'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:72:in `block in <class:Project>'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:727:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)When trying to create a new snippet, here is the exception:
"exception.class": "NoMethodError",
"exception.message": "undefined method `first' for nil:NilClass",
"exception.backtrace": [
"app/models/application_setting_implementation.rb:314:in `pick_repository_storage'",
"lib/gitlab/current_settings.rb:20:in `method_missing'",
"app/models/concerns/has_repository.rb:23:in `pick_repository_storage'",
"app/models/snippet.rb:308:in `repository_storage'",
"app/models/snippet.rb:273:in `repository'",
"app/models/concerns/has_repository.rb:43:in `repository_exists?'",
"app/services/snippets/create_service.rb:62:in `rescue in save_and_commit'",
"app/services/snippets/create_service.rb:49:in `save_and_commit'",
"app/services/snippets/create_service.rb:18:in `execute'",
"app/controllers/snippets_controller.rb:45:in `create'",
"app/controllers/application_controller.rb:497:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:488:in `set_session_storage'",
"app/controllers/application_controller.rb:482:in `set_locale'",
"lib/gitlab/error_tracking.rb:51:in `with_context'",
"app/controllers/application_controller.rb:547:in `sentry_context'",
"app/controllers/application_controller.rb:475:in `block in set_current_context'",
"lib/gitlab/application_context.rb:52:in `block in use'",
"lib/gitlab/application_context.rb:52:in `use'",
"lib/gitlab/application_context.rb:20:in `with_context'",
"app/controllers/application_controller.rb:468:in `set_current_context'"Logs from snippet migration:
{
"severity": "WARN",
"time": "2020-06-25T13:33:15.044Z",
"error_class": "NoMethodError",
"error_message": "undefined method `first' for nil:NilClass",
"context": "Job raised exception",
"class": "BackgroundMigrationWorker",
"args": [
"BackfillSnippetRepositories",
"[86, 88]"
],
"retry": 3,
"queue": "background_migration",
"jid": "e6314dbef2c0e71053283aa0",
"created_at": "2020-06-25T13:31:10.881Z",
"meta.caller_id": "BackfillSnippetRepositories",
"correlation_id": "8951b79431e98bf9d3f01d7c57e89379",
"enqueued_at": "2020-06-25T13:33:15.006Z",
"error_backtrace": [
"app/models/application_setting_implementation.rb:314:in `pick_repository_storage'",
"lib/gitlab/current_settings.rb:20:in `method_missing'",
"app/models/concerns/has_repository.rb:23:in `pick_repository_storage'",
"app/models/snippet.rb:308:in `repository_storage'",
"app/models/snippet.rb:273:in `repository'",
"lib/gitlab/background_migration/backfill_snippet_repositories.rb:27:in `block in migrate_snippets'",
"lib/gitlab/background_migration/backfill_snippet_repositories.rb:25:in `migrate_snippets'",
"lib/gitlab/background_migration/backfill_snippet_repositories.rb:13:in `perform'",
"lib/gitlab/background_migration.rb:58:in `perform'",
"app/workers/background_migration_worker.rb:31:in `block in perform'",
"app/workers/concerns/worker_context.rb:63:in `block in with_context'",
"lib/gitlab/application_context.rb:52:in `block in use'",
"lib/gitlab/application_context.rb:52:in `use'",
"app/workers/concerns/worker_context.rb:63:in `with_context'",
"app/workers/background_migration_worker.rb:27:in `perform'",
"lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
"lib/gitlab/with_request_store.rb:10:in `with_request_store'",
"lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"
]
}Any ideas?
Edited by jedis