Migration from 11.6 to 11.7 fails
<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=26644) </details> <!--IssueSummary end--> ### Summary Migration from 11.6 to 11.7 fails with a `PG::NotNullViolation`. I didn't have any issue with the upgrade from 11.5 to 11.6 and 11.6 starts and works without problems. ### Steps to reproduce The error happens on my instance but I can't reproduce it on a fresh instance. I'm using the `gitlab/gitlab-ce` docker image. ### Example Project \- ### What is the current *bug* behavior? Migration fails. ### What is the expected *correct* behavior? Migration should succeed. ### Relevant logs and/or screenshots ``` There was an error running gitlab-ctl reconfigure: bash[migrate gitlab-rails database] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" "/tmp/chef-script20190122-23-1hszz3q" ---- STDOUT: rake aborted! StandardError: An error has occurred, this and all later migrations canceled: PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint DETAIL: Failing row contains (null, null, Throughput, sum(label_replace(rate(nginx_ingress_controller_requests{namespa..., Requests / Sec, req / sec, Status Code, -6, 2019-01-22 22:03:31.421222+00, 2019-01-22 22:03:31.421222+00, t, response_metrics_nginx_ingress_16_throughput_status_code). : INSERT INTO "prometheus_metrics" ("title", "query", "y_label", "unit", "legend", "group", "created_at", "updated_at", "common", "identifier") VALUES ('Throughput', 'sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code)', 'Requests / Sec', 'req / sec', 'Status Code', -6, '2019-01-22 22:03:31.421222', '2019-01-22 22:03:31.421222', 't', 'response_metrics_nginx_ingress_16_throughput_status_code') RETURNING "id" /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:49:in `block in execute' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:87:in `process_metric_query' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:77:in `block in process_metric' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `process_metric' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:67:in `block in process_group' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `process_group' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:57:in `block in process_content' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `process_content' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:47:in `execute' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181006004100_import_common_metrics_nginx_vts.rb:9:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Caused by: ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint DETAIL: Failing row contains (null, null, Throughput, sum(label_replace(rate(nginx_ingress_controller_requests{namespa..., Requests / Sec, req / sec, Status Code, -6, 2019-01-22 22:03:31.421222+00, 2019-01-22 22:03:31.421222+00, t, response_metrics_nginx_ingress_16_throughput_status_code). : INSERT INTO "prometheus_metrics" ("title", "query", "y_label", "unit", "legend", "group", "created_at", "updated_at", "common", "identifier") VALUES ('Throughput', 'sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code)', 'Requests / Sec', 'req / sec', 'Status Code', -6, '2019-01-22 22:03:31.421222', '2019-01-22 22:03:31.421222', 't', 'response_metrics_nginx_ingress_16_throughput_status_code') RETURNING "id" /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:49:in `block in execute' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:87:in `process_metric_query' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:77:in `block in process_metric' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `process_metric' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:67:in `block in process_group' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `process_group' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:57:in `block in process_content' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `process_content' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:47:in `execute' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181006004100_import_common_metrics_nginx_vts.rb:9:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Caused by: PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint DETAIL: Failing row contains (null, null, Throughput, sum(label_replace(rate(nginx_ingress_controller_requests{namespa..., Requests / Sec, req / sec, Status Code, -6, 2019-01-22 22:03:31.421222+00, 2019-01-22 22:03:31.421222+00, t, response_metrics_nginx_ingress_16_throughput_status_code). /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:49:in `block in execute' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:87:in `process_metric_query' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:77:in `block in process_metric' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `process_metric' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:67:in `block in process_group' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `process_group' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:57:in `block in process_content' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `map' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `process_content' /opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:47:in `execute' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181006004100_import_common_metrics_nginx_vts.rb:9:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => db:migrate (See full trace by running task with --trace) == 20181006004100 ImportCommonMetricsNginxVts: migrating ====================== STDERR: ---- End output of "bash" "/tmp/chef-script20190122-23-1hszz3q" ---- Ran "bash" "/tmp/chef-script20190122-23-1hszz3q" returned 1 Running handlers complete Chef Client failed. 97 resources updated in 01 minutes 54 seconds ``` ### Output of checks #### Results of GitLab environment info ``` System information System: Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.3 Go Version: unknown GitLab information Version: 11.6.9 Revision: c46c62c Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 8.4.3 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git ``` #### Results of GitLab application Check ``` Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 8.4.3 ? ... OK (8.4.3) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/3 ... yes 1/5 ... yes 1/6 ... yes 1/7 ... yes 4/8 ... yes 5/9 ... yes 4/10 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 6 Checking GitLab App ... Finished Checking GitLab subtasks ... Finished ``` ### Possible fixes \-
issue