Review Apps initial seeding is broken due to `runners_registration_token` / `OpenSSL::Cipher::CipherError` error somehow
I found that the initial seeding of Review Apps is broken, and by looking at the migrations
job logs, there's an OpenSSL::Cipher::CipherError
error:
I have the feeling this is a side-effect of https://gitlab.com/gitlab-org/gitlab-ce/issues/59162:
- The migrations job runs the
gitlab:db:configure
task: https://gitlab.com/gitlab-org/build/CNG/blob/f3334d7e1c89783898e3cebadf2dff18afe7eb90/gitlab-rails/scripts/db-migrate#L29 - If there are no DB tables (initial migration), we load the schema and seed the database: https://gitlab.com/gitlab-org/gitlab-ce/blob/f4e8f6c89b88990d88d4ce6fddcb1b95a4cc89cd/lib/tasks/gitlab/db.rake#L58-61
- Because of https://gitlab.com/gitlab-org/gitlab-ce/issues/59162, we might encounter a unique record error when trying to create the
ApplicationSettings
singleton: seeding is aborted in that case, meaning that the following seeding scripts won't be run: no admin user created and no runners registration token created (leading to theOpenSSL::Cipher::CipherError
error).
Then when the Review Apps is redeployed:
- Since the DB has tables, we only run
db:migrate
: https://gitlab.com/gitlab-org/gitlab-ce/blob/f4e8f6c89b88990d88d4ce6fddcb1b95a4cc89cd/lib/tasks/gitlab/db.rake#L56
Hopefully https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26319 will fix everything.
Loading production environment (Rails 5.0.7.1)
irb(main):001:0> ApplicationSetting.current
=> #<ApplicationSetting id: 1, default_projects_limit: 100000, signup_enabled: true, gravatar_enabled: true, sign_in_text: nil, created_at: "2019-03-18 11:52:57", updated_at: "2019-03-18 11:52:57", home_page_url: nil, default_branch_protection: 2, restricted_visibility_levels: [], version_check_enabled: true, max_attachment_size: 10, default_project_visibility: 0, default_snippet_visibility: 0, domain_whitelist: [], user_oauth_applications: true, after_sign_out_path: nil, session_expire_delay: 10080, import_sources: ["github", "bitbucket", "bitbucket_server", "gitlab", "google_code", "fogbugz", "git", "gitlab_project", "gitea", "manifest"], help_page_text: nil, admin_notification_email: nil, shared_runners_enabled: true, max_artifacts_size: 100, runners_registration_token: nil, max_pages_size: 100, require_two_factor_authentication: false, two_factor_grace_period: 48, metrics_enabled: false, metrics_host: "localhost", metrics_pool_size: 16, metrics_timeout: 10, metrics_method_call_threshold: 10, recaptcha_enabled: false, recaptcha_site_key: nil, recaptcha_private_key: nil, metrics_port: 8089, akismet_enabled: false, akismet_api_key: nil, metrics_sample_interval: 15, sentry_enabled: false, sentry_dsn: nil, email_author_in_body: false, default_group_visibility: 0, repository_checks_enabled: true, shared_runners_text: nil, metrics_packet_size: 1, disabled_oauth_sign_in_sources: [], health_check_access_token: "<REDACTED BUT TOKEN IS PRESENT HERE>", send_user_confirmation_email: false, container_registry_token_expire_delay: 5, after_sign_up_text: nil, user_default_external: false, repository_storages: ["default"], enabled_git_access_protocol: nil, domain_blacklist_enabled: false, domain_blacklist: [], usage_ping_enabled: false, sign_in_text_html: "", help_page_text_html: "", shared_runners_text_html: "", after_sign_up_text_html: "", rsa_key_restriction: 0, dsa_key_restriction: 0, ecdsa_key_restriction: 0, ed25519_key_restriction: 0, housekeeping_enabled: true, housekeeping_bitmaps_enabled: true, housekeeping_incremental_repack_period: 10, housekeeping_full_repack_period: 50, housekeeping_gc_period: 200, html_emails_enabled: true, plantuml_url: nil, plantuml_enabled: false, terminal_max_session_time: 0, unique_ips_limit_per_user: 10, unique_ips_limit_time_window: 3600, unique_ips_limit_enabled: false, default_artifacts_expire_in: "30 days", uuid: "35cdadd1-d0b7-4f57-800e-12e175855cd5", polling_interval_multiplier: 0.1e1, cached_markdown_version: 917504, clientside_sentry_enabled: false, clientside_sentry_dsn: nil, prometheus_metrics_enabled: true, help_page_hide_commercial_content: false, help_page_support_url: nil, performance_bar_allowed_group_id: nil, hashed_storage_enabled: false, project_export_enabled: true, auto_devops_enabled: true, throttle_unauthenticated_enabled: false, throttle_unauthenticated_requests_per_period: 3600, throttle_unauthenticated_period_in_seconds: 3600, throttle_authenticated_api_enabled: false, throttle_authenticated_api_requests_per_period: 7200, throttle_authenticated_api_period_in_seconds: 3600, throttle_authenticated_web_enabled: false, throttle_authenticated_web_requests_per_period: 7200, throttle_authenticated_web_period_in_seconds: 3600, password_authentication_enabled_for_web: true, password_authentication_enabled_for_git: true, gitaly_timeout_default: 55, gitaly_timeout_medium: 30, gitaly_timeout_fast: 10, authorized_keys_enabled: true, auto_devops_domain: nil, pages_domain_verification_enabled: true, user_default_internal_regex: nil, allow_local_requests_from_hooks_and_services: false, enforce_terms: false, mirror_available: true, hide_third_party_offers: false, instance_statistics_visibility_private: false, web_ide_clientside_preview_enabled: false, user_show_add_ssh_key_message: true, usage_stats_set_by_user_id: nil, receive_max_input_size: nil, diff_max_patch_bytes: 102400, archive_builds_in_seconds: nil, commit_email_hostname: "users.noreply.gitlab-review-pravi-gitl-xnpgu0.ce.g...", protected_ci_variables: false, runners_registration_token_encrypted: "<REDACTED BUT TOKEN IS PRESENT HERE>", local_markdown_version: 0, first_day_of_week: 0>
irb(main):002:0> ApplicationSetting.current.runners_registration_token
Traceback (most recent call last):
16: from /srv/gitlab/bin/rails:4:in `<main>'
15: from /srv/gitlab/bin/rails:4:in `require'
14: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/commands.rb:18:in `<top (required)>'
13: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
12: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/commands/commands_tasks.rb:78:in `console'
11: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/commands/console_helper.rb:9:in `start'
10: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/commands/console.rb:65:in `start'
9: from (irb):2
8: from /srv/gitlab/app/models/application_setting.rb:416:in `runners_registration_token'
7: from /srv/gitlab/app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
6: from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
5: from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
4: from /srv/gitlab/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
3: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
2: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
1: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
OpenSSL::Cipher::CipherError ()
The problem happens during the initial seeding:
- https://gitlab.com/gitlab-org/build/CNG/blob/f3334d7e1c89783898e3cebadf2dff18afe7eb90/gitlab-rails/scripts/db-migrate#L36
- https://gitlab.com/gitlab-org/build/CNG/blob/f3334d7e1c89783898e3cebadf2dff18afe7eb90/gitlab-rails/scripts/disable-auth-keys-write#L15
- If I try this exact command in a Rails console, I get
irb(main):009:0> (current_settings || ::ApplicationSetting.create_from_defaults).update_attribute(:authorized_keys_enabled, false)
Traceback (most recent call last):
16: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in `each'
15: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in `block in call'
14: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:170:in `block in halting'
13: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:769:in `block in deprecated_false_terminator'
12: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:769:in `catch'
11: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:770:in `block (2 levels) in deprecated_false_terminator'
10: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:169:in `block (2 levels) in halting'
9: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:382:in `block in make_lambda'
8: from /srv/gitlab/app/models/concerns/token_authenticatable.rb:40:in `block in add_authentication_token_field'
7: from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
6: from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
5: from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
4: from /srv/gitlab/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
3: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
2: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
1: from /srv/gitlab/vendor/bundle/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
OpenSSL::Cipher::CipherError ()
It looks like ApplicationSetting.current.runners_registration_token
is nil
but ApplicationSetting.current.runners_registration_token_encrypted
isn't...
This looks similar to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4163, https://gitlab.com/gitlab-org/gitlab-ce/issues/56171. Also, this relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/56566.
Could it be because ENV['GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN']
was nil
?
I will investigate this further.