Skip to content

Broken migration due to not-null constraint

Summary

Updating from GitLab CE v10.7.1 to GitLab CE v10.8.1 using the Omnibus package results in a database migration failure.

Steps to reproduce

Upgrade GitLab CE to v10.8.1 and run sudo -H gitlab-ctl reconfigue (though dpkg will typically run this automatically). There are presumably some specifics of our installation which trigger the issue for us but not others. Our GitLab installation has gone through many updates dating back years.

What is the current bug behavior?

A database migration failure occurs during upgrade while applying 20180413022611_create_missing_namespace_for_internal_users.rb. The issue is caused by an attempted insertion of a row to the namespaces table without setting the created_at and updated_at fields. This results in a not-null constraint violation.

What is the expected correct behavior?

Migration runs successfully.

Relevant logs and/or screenshots

There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20180528-60716-o28qkq" ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::NotNullViolation: ERROR:  null value in column "created_at" violates not-null constraint
DETAIL:  Failing row contains (54, ghost, ghost, 52, null, null, null, , null, f, 20, f, null, null, null, f, 48, null, null).
: INSERT INTO namespaces(owner_id, path, name, created_at) VALUES(52, 'ghost', 'ghost', NOW()) RETURNING "id"
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:49:in `create_namespace'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:6:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR:  null value in column "created_at" violates not-null constraint
DETAIL:  Failing row contains (54, ghost, ghost, 52, null, null, null, , null, f, 20, f, null, null, null, f, 48, null, null).
: INSERT INTO namespaces(owner_id, path, name, created_at) VALUES(52, 'ghost', 'ghost', NOW()) RETURNING "id"
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:49:in `create_namespace'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:6:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::NotNullViolation: ERROR:  null value in column "created_at" violates not-null constraint
DETAIL:  Failing row contains (54, ghost, ghost, 52, null, null, null, , null, f, 20, f, null, null, null, f, 48, null, null).
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:49:in `create_namespace'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:6:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180413022611_create_missing_namespace_for_internal_users.rb:5:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20180413022611 CreateMissingNamespaceForInternalUsers: migrating ===========
-- column_exists?(:users, :support_bot)
   -> 0.0021s
STDERR:
---- End output of "bash"  "/tmp/chef-script20180528-60716-o28qkq" ----
Ran "bash"  "/tmp/chef-script20180528-60716-o28qkq" returned 1

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.7p456
Gem Version:    2.6.14
Bundler Version:1.13.7
Rake Version:   12.3.1
Redis Version:  3.2.11
Git Version:    2.16.3
Sidekiq Version:5.0.5
Go Version:     unknown

GitLab information
Version:        10.8.1
Revision:       21a8d61
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.corp..com
HTTP Clone URL: https://gitlab.corp..com/some-group/some-project.git
SSH Clone URL:  git@git.corp..com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        7.1.2
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...

GitLab Shell version >= 7.1.2 ? ... OK (7.1.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/3 ... ok
5/6 ... ok
5/7 ... ok
5/8 ... ok
5/9 ... ok
5/11 ... ok
5/13 ... ok
5/14 ... ok
7/15 ... ok
7/16 ... ok
8/17 ... ok
8/18 ... ok
17/19 ... ok
9/20 ... ok
9/21 ... ok
5/22 ... ok
5/23 ... ok
5/24 ... ok
17/25 ... ok
5/26 ... ok
17/27 ... ok
4/28 ... ok
8/29 ... ok
9/30 ... ok
5/31 ... ok
17/32 ... ok
5/34 ... ok
5/35 ... ok
16/36 ... ok
3/37 ... ok
17/38 ... ok
12/39 ... ok
5/43 ... ok
5/45 ... ok
3/46 ... ok
5/49 ... ok
5/50 ... ok
27/51 ... ok
17/52 ... ok
17/53 ... ok
5/55 ... ok
6/56 ... ok
6/57 ... ok
5/59 ... ok
12/60 ... ok
3/61 ... ok
4/62 ... ok
5/63 ... ok
5/64 ... ok
5/65 ... ok
5/66 ... ok
6/67 ... ok
3/68 ... ok
9/69 ... ok
5/70 ... ok
6/71 ... ok
29/72 ... ok
29/73 ... ok
11/74 ... repository is empty
3/75 ... ok
8/77 ... ok
5/79 ... ok
6/80 ... ok
4/81 ... ok
7/82 ... ok
5/83 ... ok
5/84 ... ok
8/85 ... ok
5/87 ... ok
12/88 ... ok
12/90 ... ok
27/91 ... ok
32/92 ... repository is empty
12/97 ... ok
17/98 ... ok
5/99 ... ok
3/100 ... ok
17/101 ... ok
12/103 ... ok
17/104 ... ok
12/106 ... ok
8/108 ... ok
27/109 ... ok
27/110 ... ok
8/111 ... ok
17/112 ... ok
8/113 ... ok
8/114 ... ok
8/115 ... ok
12/116 ... ok
12/117 ... ok
17/118 ... ok
27/119 ... ok
17/120 ... ok
8/121 ... ok
12/122 ... ok
17/124 ... ok
17/125 ... ok
5/126 ... repository is empty
5/127 ... ok
37/128 ... ok
12/129 ... ok
12/130 ... ok
7/131 ... ok
17/132 ... ok
17/133 ... ok
17/134 ... ok
12/135 ... ok
5/137 ... ok
45/138 ... ok
17/139 ... ok
8/141 ... ok
8/142 ... ok
27/143 ... ok
2/144 ... ok
8/146 ... ok
17/147 ... ok
17/148 ... ok
47/149 ... ok
12/150 ... ok
47/151 ... ok
48/152 ... ok
48/153 ... ok
48/154 ... ok
48/155 ... ok
5/157 ... ok
8/158 ... ok
8/159 ... ok
5/160 ... ok
8/161 ... ok
45/162 ... ok
50/163 ... ok
2/164 ... ok
2/165 ... ok
5/166 ... ok
50/167 ... ok
50/168 ... ok
8/169 ... ok
50/170 ... ok
5/171 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


Server: ldapmain
not verifying SSL hostname of LDAPS server '.com:389'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)



Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/3 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
5/9 ... yes
5/11 ... yes
5/13 ... yes
5/14 ... yes
7/15 ... yes
7/16 ... yes
8/17 ... yes
8/18 ... yes
17/19 ... yes
9/20 ... yes
9/21 ... yes
5/22 ... yes
5/23 ... yes
5/24 ... yes
17/25 ... yes
5/26 ... yes
17/27 ... yes
4/28 ... yes
8/29 ... yes
9/30 ... yes
5/31 ... yes
17/32 ... yes
5/34 ... yes
5/35 ... yes
16/36 ... yes
3/37 ... yes
17/38 ... yes
12/39 ... yes
5/43 ... yes
5/45 ... yes
3/46 ... yes
5/49 ... yes
5/50 ... yes
27/51 ... yes
17/52 ... yes
17/53 ... yes
5/55 ... yes
6/56 ... yes
6/57 ... yes
5/59 ... yes
12/60 ... yes
3/61 ... yes
4/62 ... yes
5/63 ... yes
5/64 ... yes
5/65 ... yes
5/66 ... yes
6/67 ... yes
3/68 ... yes
9/69 ... yes
5/70 ... yes
6/71 ... yes
29/72 ... yes
29/73 ... yes
11/74 ... yes
3/75 ... yes
8/77 ... yes
5/79 ... yes
6/80 ... yes
4/81 ... yes
7/82 ... yes
5/83 ... yes
5/84 ... yes
8/85 ... yes
5/87 ... yes
12/88 ... yes
12/90 ... yes
27/91 ... yes
32/92 ... yes
12/97 ... yes
17/98 ... yes
5/99 ... yes
3/100 ... yes
17/101 ... yes
12/103 ... yes
17/104 ... yes
12/106 ... yes
8/108 ... yes
27/109 ... yes
27/110 ... yes
8/111 ... yes
17/112 ... yes
8/113 ... yes
8/114 ... yes
8/115 ... yes
12/116 ... yes
12/117 ... yes
17/118 ... yes
27/119 ... yes
17/120 ... yes
8/121 ... yes
12/122 ... yes
17/124 ... yes
17/125 ... yes
5/126 ... yes
5/127 ... yes
37/128 ... yes
12/129 ... yes
12/130 ... yes
7/131 ... yes
17/132 ... yes
17/133 ... yes
17/134 ... yes
12/135 ... yes
5/137 ... yes
45/138 ... yes
17/139 ... yes
8/141 ... yes
8/142 ... yes
27/143 ... yes
2/144 ... yes
8/146 ... yes
17/147 ... yes
17/148 ... yes
47/149 ... yes
12/150 ... yes
47/151 ... yes
48/152 ... yes
48/153 ... yes
48/154 ... yes
48/155 ... yes
5/157 ... yes
8/158 ... yes
8/159 ... yes
5/160 ... yes
8/161 ... yes
45/162 ... yes
50/163 ... yes
2/164 ... yes
2/165 ... yes
5/166 ... yes
50/167 ... yes
50/168 ... yes
8/169 ... yes
50/170 ... yes
5/171 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.7)
Git version >= 2.9.5 ? ... yes (2.16.3)
Git user has default SSH configuration? ... yes
Active users: ... 19


Checking GitLab ... Finished

Possible fixes

The problem line in the migration appears to be here. We were able to workaround the issue by modifying it as follows:

48c48
<     insert_query = "INSERT INTO namespaces(owner_id, path, name, created_at, updated_at) VALUES(#{user_id}, '#{path}', '#{path}', NOW(), NOW())"
---
>     insert_query = "INSERT INTO namespaces(owner_id, path, name) VALUES(#{user_id}, '#{path}', '#{path}')"