Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Snippets
  • Sign up now
  • Login
  • Sign in / Register
  • omnibus-gitlab omnibus-gitlab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 1,096
    • Issues 1,096
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 82
    • Merge requests 82
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • omnibus-gitlabomnibus-gitlab
  • Issues
  • #5612
Closed
Open
Issue created Aug 29, 2020 by Kumar@interviewer

Gitlab upgrade issue today: reconfigure - ERROR: permission denied to create extension "btree_gist"

Please also have a look at gitlab#242388 (closed) Related to : gitlab-org/charts/gitlab#2113 (closed)

Summary

When you run apt-get update / upgrade on a production system today, you get into this problem

Steps to reproduce

I am not sure I can reproduce this bug without installing a new version.

What is the current bug behavior?

apt-get update && apt upgrade && gitlab-reconfigure pushed out this output:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] == 20200504191813 AddFingerprintToEvents: migrating ===========================
              -- column_exists?(:events, :fingerprint)
                 -> 0.0030s
              -- add_column(:events, :fingerprint, :binary)
                 -> 0.0095s
              -- transaction_open?()
                 -> 0.0000s
              -- execute("ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n")
                 -> 0.0100s
              -- execute("ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;")
                 -> 0.0162s
              == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================
              
              == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
              -- transaction_open?()
                 -> 0.0000s
              -- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
                 -> 0.0113s
              -- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
                 -> 0.0216s
              == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 
              
              == 20200515152649 EnableBtreeGistExtension: migrating =========================
              -- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
              rake aborted!
              StandardError: An error has occurred, this and all later migrations canceled:
              
              PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
              HINT:  Must be superuser to create this extension.
              /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              
              Caused by:
              ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
              HINT:  Must be superuser to create this extension.
              /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              
              Caused by:
              PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
              HINT:  Must be superuser to create this extension.
              /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => db:migrate
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
    STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
    -- column_exists?(:events, :fingerprint)
       -> 0.0030s
    -- add_column(:events, :fingerprint, :binary)
       -> 0.0095s
    -- transaction_open?()
       -> 0.0000s
    -- execute("ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n")
       -> 0.0100s
    -- execute("ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;")
       -> 0.0162s
    == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================
    
    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
    -- transaction_open?()
       -> 0.0000s
    -- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
       -> 0.0113s
    -- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
       -> 0.0216s
    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 
    
    == 20200515152649 EnableBtreeGistExtension: migrating =========================
    -- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
    rake aborted!
    StandardError: An error has occurred, this and all later migrations canceled:
    
    PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
    HINT:  Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    
    Caused by:
    ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
    HINT:  Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    
    Caused by:
    PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
    HINT:  Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
    Ran "bash"  "/tmp/chef-script20200829-1833-1bujtkw" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     55: bash "migrate gitlab-rails database" do
     56:   code <<-EOH
     57:     set -e
     58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     59:     umask 077
     60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     61:     STATUS=${PIPESTATUS[0]}
     62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
     63:     echo $STATUS > #{db_migrate_status_file}
     64:     exit $STATUS
     65:   EOH
     66:   environment env_variables unless env_variables.empty?
     67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     68:   dependent_services.each do |svc|
     69:     notifies :restart, svc, :immediately
     70:   end
     71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     73: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49\n    exit $STATUS\n"
      domain nil
      user nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49 | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=15.12.22
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    
Recipe: gitlab::puma
  * runit_service[puma] action restart (up to date)
  * runit_service[puma] action restart (up to date)
Recipe: gitlab::sidekiq-cluster
  * sidekiq_service[sidekiq] action restart
  Recipe: <Dynamically Defined Resource>
    * service[sidekiq] action nothing (skipped due to action :nothing)
    * runit_service[sidekiq] action restart (up to date)
     (up to date)
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    - execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
  * runit_service[gitaly] action hup
    - send hup to runit_service[gitaly]

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
-- column_exists?(:events, :fingerprint)
   -> 0.0030s
-- add_column(:events, :fingerprint, :binary)
   -> 0.0095s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n")
   -> 0.0100s
-- execute("ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;")
   -> 0.0162s
== 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
   -> 0.0113s
-- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
   -> 0.0216s
== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 

== 20200515152649 EnableBtreeGistExtension: migrating =========================
-- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
HINT:  Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
HINT:  Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
HINT:  Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200829-1833-1bujtkw" ----
Ran "bash"  "/tmp/chef-script20200829-1833-1bujtkw" returned 1

Running handlers complete
Chef Infra Client failed. 13 resources updated in 01 minutes 06 seconds
Expand for output related to GitLab environment info


System information
System:   Ubuntu 18.04
Current User: git
Using RVM:  no
Ruby Version: 2.6.6p146
Gem Version:  2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version:  5.0.9
Git Version:  2.28.0
Sidekiq Version:5.2.9
Go Version: unknown

GitLab information
Version:  13.3.2
Revision: f4a969f7f49
Directory:  /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.4
URL:    https://git.mydomain.com
HTTP Clone URL: https://git.mydomain.com/some-group/some-project.git
SSH Clone URL:  git@git.mydomain.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:  13.6.0
Repository storage paths:
- default:  /home/git/repositories
GitLab Shell path:    /opt/gitlab/embedded/service/gitlab-shell
Git:    /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/6 ... yes 7/7 ... yes 16/9 ... yes 16/10 ... yes 7/11 ... yes 16/19 ... yes 7/20 ... yes 7/21 ... yes 16/23 ... yes 16/24 ... yes 16/26 ... yes 16/27 ... yes 16/28 ... yes 16/29 ... yes 16/30 ... yes 16/31 ... yes 14/32 ... yes 16/33 ... yes 10/35 ... yes 16/36 ... yes 1/38 ... yes 1/40 ... yes 10/41 ... yes 14/42 ... yes 16/43 ... yes 16/44 ... yes 16/45 ... yes 14/46 ... yes 16/48 ... yes 16/49 ... yes 24/50 ... yes 16/51 ... yes 16/52 ... yes 1/53 ... yes 1/54 ... yes 1/55 ... yes 30/56 ... yes 30/58 ... yes 31/59 ... yes 31/61 ... yes 31/62 ... yes 32/63 ... yes 28/64 ... yes 32/65 ... yes 32/66 ... yes 30/67 ... yes 33/68 ... yes 26/69 ... yes 1/70 ... yes 32/71 ... yes 28/72 ... yes 32/73 ... yes 32/75 ... yes 32/76 ... yes 1/79 ... yes 38/80 ... yes 38/81 ... yes 1/82 ... yes 38/83 ... yes 38/84 ... yes 32/85 ... yes 32/86 ... yes 1/87 ... yes 1/88 ... yes 39/89 ... yes 32/90 ... yes 32/91 ... yes 32/94 ... yes 32/95 ... yes 1/96 ... yes 32/97 ... yes 1/98 ... yes 1/99 ... yes 43/100 ... yes 1/101 ... yes 43/102 ... yes 45/103 ... yes 38/104 ... yes 43/105 ... yes 45/107 ... yes 37/108 ... yes 32/109 ... yes 1/110 ... yes 32/111 ... yes 32/112 ... yes 1/113 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 11 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Edited Aug 29, 2020 by Kumar
Assignee
Assign to
Time tracking