Skip to content

gitlab-ctl reconfigure triggers 'Peer authentication failed for user "gitlab"' errors in PostgreSQL log

Summary

Working with a customer to fix upgrade issues, they encountered the following errors in /var/log/gitlab/postgresql/current.

The errors are generated by gitlab-ctl reconfigure

DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
FATAL:  Peer authentication failed for user "gitlab"

Steps to reproduce

  1. Configure a single Omnibus node as attached; you'll get Gitaly, PostgreSQL, and Redis.
  2. Run gitlab-ctl reconfigure

Chef/CINC appears to be making requests to PostgreSQL that are denied.

What is the current bug behavior?

configuration related authentication failure on a Omnibus PostgreSQL install.

What is the expected correct behavior?

authentication succeeds (or the operation isn't performed)

Relevant logs

Relevant logs

Details of package version

customer is running 16.0.9, observed on this Support Engineer's test environment: gitlab-ee-16.3.7-ee.0.el7.x86_64

Environment details

My test environment

  • Operating System: Centos7

  • Installation Target, remove incorrect values:

    • VM: Other KVM

  • Installation Type, remove incorrect values:

    • Upgrade from version n/a
    • Other: Two node install; backend plus frontend, so database traffic runs over the network. Customer is a scaled hybrid deployment - PgBouncer, load balancers etc., but the minimum change seems to be switching from socket to TCP traffic for PostgreSQL

  • Is there any other software running on the machine: no

  • Is this a single or multiple node installation?

  • Resources

    my test system.

    • CPU: 1
    • Memory total: 2gb

Configuration details

All relevant configuration for PostgreSQL seems to be the same as our reference architectures

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
postgresql['listen_address'] = '0.0.0.0'
postgresql['port'] = 5432
postgresql['sql_user_password'] = File.open('/etc/gitlab/secrets/postgresql_hash').first.chomp
postgresql['trust_auth_cidr_addresses'] = ['192.168.1.0/24','127.0.0.1/32']
gitaly['enable'] = true
gitlab_shell['secret_token'] = File.open('/etc/gitlab/secrets/gitlab_shell_token').first.chomp
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
gitaly['configuration'] = {
listen_addr: "0.0.0.0:8075",
auth: {
token: File.open('/etc/gitlab/secrets/gitaly_token').first.chomp
},
storage: [
{
name: 'default',
path: '/var/opt/gitlab/git-data/repositories',
},
],
}
redis['enable'] = true
redis['bind'] = '0.0.0.0'
redis['port'] = 6379
redis['password'] = File.read('/etc/gitlab/secrets/redis_password').delete!("\n")
prometheus_monitoring['enable'] = false
puma['enable'] = false
nginx['enable'] = false
registry['enable'] = false
gitlab_workhorse['enable'] = false
sidekiq['enable'] = false
gitlab_kas['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['rake_cache_clear'] = false
postgresql['sql_replication_password'] = File.open('/etc/gitlab/secrets/sql_replication_password_md5').first.chomp
roles(['geo_primary_role'])
postgresql['max_replication_slots'] = 1
postgresql['max_connections'] = 400
Edited by Ben Prescott_