Skip to content

GitLab Next

Open
Created by David@interviewer

Gitlab upgrade issue today: reconfigure - ERROR: permission denied to create extension "btree_gist"

Please also have a look at gitlab#242388 (closed) Related to : gitlab-org/charts/gitlab#2113 (closed)

Summary

When you run apt-get update / upgrade on a production system today, you get into this problem

Steps to reproduce

I am not sure I can reproduce this bug without installing a new version.

What is the current bug behavior?

apt-get update && apt upgrade && gitlab-reconfigure pushed out this output:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] == 20200504191813 AddFingerprintToEvents: migrating ===========================
              -- column_exists?(:events, :fingerprint)
                 -> 0.0030s
              -- add_column(:events, :fingerprint, :binary)
                 -> 0.0095s
              -- transaction_open?()
                 -> 0.0000s
              -- execute("ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n")
                 -> 0.0100s
              -- execute("ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;")
                 -> 0.0162s
              == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================
              
              == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
              -- transaction_open?()
                 -> 0.0000s
              -- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
                 -> 0.0113s
              -- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
                 -> 0.0216s
              == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 
              
              == 20200515152649 EnableBtreeGistExtension: migrating =========================
              -- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
              rake aborted!
              StandardError: An error has occurred, this and all later migrations canceled:
              
              PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
              HINT:  Must be superuser to create this extension.
              /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              
              Caused by:
              ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
              HINT:  Must be superuser to create this extension.
              /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              
              Caused by:
              PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
              HINT:  Must be superuser to create this extension.
              /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => db:migrate
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
    STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
    -- column_exists?(:events, :fingerprint)
       -> 0.0030s
    -- add_column(:events, :fingerprint, :binary)
       -> 0.0095s
    -- transaction_open?()
       -> 0.0000s
    -- execute("ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n")
       -> 0.0100s
    -- execute("ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;")
       -> 0.0162s
    == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================
    
    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
    -- transaction_open?()
       -> 0.0000s
    -- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
       -> 0.0113s
    -- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
       -> 0.0216s
    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 
    
    == 20200515152649 EnableBtreeGistExtension: migrating =========================
    -- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
    rake aborted!
    StandardError: An error has occurred, this and all later migrations canceled:
    
    PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
    HINT:  Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    
    Caused by:
    ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
    HINT:  Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    
    Caused by:
    PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
    HINT:  Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
    Ran "bash"  "/tmp/chef-script20200829-1833-1bujtkw" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     55: bash "migrate gitlab-rails database" do
     56:   code <<-EOH
     57:     set -e
     58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     59:     umask 077
     60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     61:     STATUS=${PIPESTATUS[0]}
     62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
     63:     echo $STATUS > #{db_migrate_status_file}
     64:     exit $STATUS
     65:   EOH
     66:   environment env_variables unless env_variables.empty?
     67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     68:   dependent_services.each do |svc|
     69:     notifies :restart, svc, :immediately
     70:   end
     71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     73: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49\n    exit $STATUS\n"
      domain nil
      user nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49 | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=15.12.22
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    
Recipe: gitlab::puma
  * runit_service[puma] action restart (up to date)
  * runit_service[puma] action restart (up to date)
Recipe: gitlab::sidekiq-cluster
  * sidekiq_service[sidekiq] action restart
  Recipe: <Dynamically Defined Resource>
    * service[sidekiq] action nothing (skipped due to action :nothing)
    * runit_service[sidekiq] action restart (up to date)
     (up to date)
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    - execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
  * runit_service[gitaly] action hup
    - send hup to runit_service[gitaly]

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
-- column_exists?(:events, :fingerprint)
   -> 0.0030s
-- add_column(:events, :fingerprint, :binary)
   -> 0.0095s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n")
   -> 0.0100s
-- execute("ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;")
   -> 0.0162s
== 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
   -> 0.0113s
-- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
   -> 0.0216s
== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 

== 20200515152649 EnableBtreeGistExtension: migrating =========================
-- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
HINT:  Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
HINT:  Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
HINT:  Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
Ran "bash"  "/tmp/chef-script20200829-1833-1bujtkw" returned 1

Running handlers complete
Chef Infra Client failed. 13 resources updated in 01 minutes 06 seconds
Expand for output related to GitLab environment info 


System information
System:   Ubuntu 18.04
Current User: git
Using RVM:  no
Ruby Version: 2.6.6p146
Gem Version:  2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version:  5.0.9
Git Version:  2.28.0
Sidekiq Version:5.2.9
Go Version: unknown

GitLab information
Version:  13.3.2
Revision: f4a969f7f49
Directory:  /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.4
URL:    https://git.mydomain.com
HTTP Clone URL: https://git.mydomain.com/some-group/some-project.git
SSH Clone URL:  git@git.mydomain.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:  13.6.0
Repository storage paths:
- default:  /home/git/repositories
GitLab Shell path:    /opt/gitlab/embedded/service/gitlab-shell
Git:    /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/6 ... yes
7/7 ... yes
16/9 ... yes
16/10 ... yes
7/11 ... yes
16/19 ... yes
7/20 ... yes
7/21 ... yes
16/23 ... yes
16/24 ... yes
16/26 ... yes
16/27 ... yes
16/28 ... yes
16/29 ... yes
16/30 ... yes
16/31 ... yes
14/32 ... yes
16/33 ... yes
10/35 ... yes
16/36 ... yes
1/38 ... yes
1/40 ... yes
10/41 ... yes
14/42 ... yes
16/43 ... yes
16/44 ... yes
16/45 ... yes
14/46 ... yes
16/48 ... yes
16/49 ... yes
24/50 ... yes
16/51 ... yes
16/52 ... yes
1/53 ... yes
1/54 ... yes
1/55 ... yes
30/56 ... yes
30/58 ... yes
31/59 ... yes
31/61 ... yes
31/62 ... yes
32/63 ... yes
28/64 ... yes
32/65 ... yes
32/66 ... yes
30/67 ... yes
33/68 ... yes
26/69 ... yes
1/70 ... yes
32/71 ... yes
28/72 ... yes
32/73 ... yes
32/75 ... yes
32/76 ... yes
1/79 ... yes
38/80 ... yes
38/81 ... yes
1/82 ... yes
38/83 ... yes
38/84 ... yes
32/85 ... yes
32/86 ... yes
1/87 ... yes
1/88 ... yes
39/89 ... yes
32/90 ... yes
32/91 ... yes
32/94 ... yes
32/95 ... yes
1/96 ... yes
32/97 ... yes
1/98 ... yes
1/99 ... yes
43/100 ... yes
1/101 ... yes
43/102 ... yes
45/103 ... yes
38/104 ... yes
43/105 ... yes
45/107 ... yes
37/108 ... yes
32/109 ... yes
1/110 ... yes
32/111 ... yes
32/112 ... yes
1/113 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 11
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by David