Migration from 11.6 to 11.7 fails
Summary
Migration from 11.6 to 11.7 fails with a PG::NotNullViolation. I didn't have any issue with the upgrade from 11.5 to 11.6 and 11.6 starts and works without problems.
Steps to reproduce
The error happens on my instance but I can't reproduce it on a fresh instance. I'm using the gitlab/gitlab-ce docker image.
Example Project
-
What is the current bug behavior?
Migration fails.
What is the expected correct behavior?
Migration should succeed.
Relevant logs and/or screenshots
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20190122-23-1hszz3q" ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint
DETAIL: Failing row contains (null, null, Throughput, sum(label_replace(rate(nginx_ingress_controller_requests{namespa..., Requests / Sec, req / sec, Status Code, -6, 2019-01-22 22:03:31.421222+00, 2019-01-22 22:03:31.421222+00, t, response_metrics_nginx_ingress_16_throughput_status_code).
: INSERT INTO "prometheus_metrics" ("title", "query", "y_label", "unit", "legend", "group", "created_at", "updated_at", "common", "identifier") VALUES ('Throughput', 'sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code)', 'Requests / Sec', 'req / sec', 'Status Code', -6, '2019-01-22 22:03:31.421222', '2019-01-22 22:03:31.421222', 't', 'response_metrics_nginx_ingress_16_throughput_status_code') RETURNING "id"
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:49:in `block in execute'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:87:in `process_metric_query'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:77:in `block in process_metric'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `process_metric'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:67:in `block in process_group'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `process_group'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:57:in `block in process_content'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `process_content'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:47:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181006004100_import_common_metrics_nginx_vts.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint
DETAIL: Failing row contains (null, null, Throughput, sum(label_replace(rate(nginx_ingress_controller_requests{namespa..., Requests / Sec, req / sec, Status Code, -6, 2019-01-22 22:03:31.421222+00, 2019-01-22 22:03:31.421222+00, t, response_metrics_nginx_ingress_16_throughput_status_code).
: INSERT INTO "prometheus_metrics" ("title", "query", "y_label", "unit", "legend", "group", "created_at", "updated_at", "common", "identifier") VALUES ('Throughput', 'sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".*%{ci_environment_slug}.*"}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code)', 'Requests / Sec', 'req / sec', 'Status Code', -6, '2019-01-22 22:03:31.421222', '2019-01-22 22:03:31.421222', 't', 'response_metrics_nginx_ingress_16_throughput_status_code') RETURNING "id"
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:49:in `block in execute'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:87:in `process_metric_query'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:77:in `block in process_metric'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `process_metric'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:67:in `block in process_group'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `process_group'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:57:in `block in process_content'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `process_content'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:47:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181006004100_import_common_metrics_nginx_vts.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint
DETAIL: Failing row contains (null, null, Throughput, sum(label_replace(rate(nginx_ingress_controller_requests{namespa..., Requests / Sec, req / sec, Status Code, -6, 2019-01-22 22:03:31.421222+00, 2019-01-22 22:03:31.421222+00, t, response_metrics_nginx_ingress_16_throughput_status_code).
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:49:in `block in execute'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:87:in `process_metric_query'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:77:in `block in process_metric'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:76:in `process_metric'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:67:in `block in process_group'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:66:in `process_group'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:57:in `block in process_content'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `map'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:56:in `process_content'
/opt/gitlab/embedded/service/gitlab-rails/db/importers/common_metrics_importer.rb:47:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181006004100_import_common_metrics_nginx_vts.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20181006004100 ImportCommonMetricsNginxVts: migrating ======================
STDERR:
---- End output of "bash" "/tmp/chef-script20190122-23-1hszz3q" ----
Ran "bash" "/tmp/chef-script20190122-23-1hszz3q" returned 1
Running handlers complete
Chef Client failed. 97 resources updated in 01 minutes 54 secondsOutput of checks
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.5.3p105
Gem Version: 2.7.6
Bundler Version:1.16.6
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.18.1
Sidekiq Version:5.2.3
Go Version: unknown
GitLab information
Version: 11.6.9
Revision: c46c62c
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 8.4.3
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/gitResults of GitLab application Check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.3 ? ... OK (8.4.3)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/3 ... yes
1/5 ... yes
1/6 ... yes
1/7 ... yes
4/8 ... yes
5/9 ... yes
4/10 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 6
Checking GitLab App ... Finished
Checking GitLab subtasks ... FinishedPossible fixes
-