Upgrade generates "userdel: user 'gitlab-backup' does not exist" error

Summary

In preparation for a production upgrade, a customer upgraded their DEV environment from gitlab-ee-17.1.1-ee.0.el8.x86_64 to gitlab-ee-17.2.1-ee.0.el8.x86_64 and encountered the following error:

Infra Phase failed. 1 resources updated in 09 seconds
[2024-08-01T14:59:12+01:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-08-01T14:59:12+01:00] FATAL: ---------------------------------------------------------------------------------------
[2024-08-01T14:59:12+01:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-08-01T14:59:12+01:00] FATAL: ---------------------------------------------------------------------------------------
[2024-08-01T14:59:12+01:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: account[GitLab Backup User] (gitlab::gitlab-backup-cli_disable line 29) had an error: Mixlib::ShellOut::ShellCommandFailed: linux_user[gitlab-backup] (gitlab::gitlab-backup-cli_disable line 75) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '6'
---- Begin output of ["userdel", "gitlab-backup"] ----
STDOUT: 
STDERR: userdel: user 'gitlab-backup' does not exist
---- End output of ["userdel", "gitlab-backup"] ----
Ran ["userdel", "gitlab-backup"] returned 6

Their upgrade path was:

16.10.0 > 16.11.1 > 16.11.3 > 17.0.1 > 17.1.1 > 17.2.1

Additional info

gitlab-backup isn't a local user on their systems. So attempting to delete the user will fail :

ud006624[2]{0}$ id -a gitlab-backup
uid=77566(gitlab-backup) gid=71360(git) groups=71360(git)
ud006624[2]{0}$ grep gitlab-backup /etc/passwd
Previous exit value was: 1
ud006624[2]{1}$ getent passwd gitlab-backup
gitlab-backup:*:77566:71360:<NAME> - SCTASK0633046:/var/opt/gitlab/gitlab-backup-home:/bin/ksh

ud006624[2]{2}$ grep passwd /etc/nsswitch.conf | grep -v \#
passwd:     files sss systemd

userdel command operates on local system users:

DESCRIPTION
       The userdel command modifies the system account files, deleting all
       entries that refer to the user name LOGIN. The named user must exist.

Note: the existence of gitlab-backup as a user at all is because they created that some time back (as an LDAP user) for their PRD backup scripts. Another reason they'd quite like the upgrade not to delete it at all. But the main issue is that no previous upgrades have created this user locally, so it’s not there for the 17.2 upgrade to delete.

Environment

The DEV environment is a scaled down version of their scaled production architecture, with the same configuration. See attached:

PROD_Architecture_Doc.pdf

The gitlab-backup user doesn’t exist locally on Prod either, so they assume the upgrade would fail there too.

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs

Details of package version

Provide the package version installation details

gitlab-ee-17.2.1-ee.0.el8.x86_64

Environment details

Operating System: REPLACE-WITH-DETAILS
Installation Target, remove incorrect values:
- Bare Metal Machine
- VM: Digital Ocean, AWS, GCP, Azure, Other REPLACE-WITH-DETAILS
- Other: REPLACE-WITH-DETAILS
Installation Type, remove incorrect values:
- New Installation
- Upgrade from version REPLACE-WITH-DETAILS
- Other: REPLACE-WITH-DETAILS
Is there any other software running on the machine: REPLACE-WITH-DETAILS
Is this a single or multiple node installation?
Resources
- CPU: REPLACE-WITH-DETAILS
- Memory total: REPLACE-WITH-DETAILS

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

```
roles ['geo_primary_role']
external_url 'https://REDACTEDgitdev.bREDACTEDb.com/'
pages_external_url 'https://REDACTEDgitpagesdev.bREDACTEDb.com'
gitlab_pages['access_control'] = true
gitlab_rails['gitlab_default_projects_limit'] = 100
gitlab_rails['gitlab_username_changing_enabled'] = true
gitlab_rails['gitlab_signup_enabled'] = false
gitlab_rails['gravatar_enabled'] = true
gitlab_rails['auto_migrate'] = true
gitlab_rails['trusted_certs_dir'] = '/var/opt/gitlab/cbs-sdlc/etc-gitlab-trusted-certs'
manage_accounts['enable'] = false
gitlab_rails['gitlab_default_projects_features_visibility_level'] = 'private'
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-EOS
main:
  label: 'Active Directory'
  host: 'bREDACTEDb.com'
  port: 636
  active_directory: true
  uid: 'REDACTED ACCOUNT NAME'
  encryption: 'simple_tls'
  verify_certificates: false
  bind_dn: 'REDACTED BIND_DN'
  password: "REDACTED PASSWORD"
  base: 'DC=bREDACTEDb,DC=com'
  group_base: 'OU=Application Access,OU=groups,DC=bREDACTEDb,DC=com'
EOS
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
gitlab_rails['omniauth_sync_profile_attributes'] = ['email']
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_saml_user'] = true
gitlab_rails['omniauth_providers'] = [
  {
    name: 'saml',
    args: {
             idp_cert_fingerprint: 'REDACTED FINGERPRINT',
             idp_sso_target_url: 'https://login.microsoftonline.com/REDACTED FINGERPRINT/saml2',
             issuer: 'https://REDACTEDgitdev.bREDACTEDb.com',
             name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent',
             attribute_statements: { 
                        email: ["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"],
                        name: ["http://schemas.microsoft.com/identity/claims/name"] 
             }
           },
    label: 'BREDACTEDB Login'
  }
]
gitlab_rails['backup_keep_time'] = 259200
prometheus[ 'enable' ] = false
node_exporter['listen_address'] = '0.0.0.0:9100'
node_exporter['flags'] = {
  'collector.filesystem.ignored-fs-types' => "'^(tmpfs|autofs|devpts|devtmpfs|binfmt_misc|selinuxfs|proc|sysfs|rpc_pipefs|nfs|usbfs|rootfs|vxodmfs|cgroup|configfs|debugfs|hugetlbfs|mqueue|pstore|securityfs)$'",
  'collector.filesystem.ignored-mount-points' => "/boot",
}
gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229"
gitlab_exporter['listen_address'] = '0.0.0.0'
gitlab_exporter['listen_port'] = '9168'
puma['listen'] = '0.0.0.0'
sidekiq['listen_address'] = '0.0.0.0'
redis['enable'] = true
redis_exporter['listen_address'] = '0.0.0.0:9121'
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '10.139.54.1' , '10.23.154.143' , '10.109.202.193' ]
registry['debug_addr'] = "0.0.0.0:5001"
nginx['status'] = {
  "enable" => true,
  "listen_addresses" => ["0.0.0.0"],
  "fqdn" => "REDACTEDgitdev.bREDACTEDb.com",
  "port" => 9999,
  "vts_enable" => true,
  "options" => {
    "server_tokens" => "off", # Don't show the version of NGINX
    "access_log" => "off", # Disable logs for stats
    "allow" => "0.0.0.0/0", # Only allow access from localhost
  }
}
 postgresql['enable'] = false
 postgresql['version'] = 14
 gitlab_rails['db_adapter'] = "postgresql"
 gitlab_rails['db_encoding'] = "unicode"
 gitlab_rails['db_database'] = "gitlabhq_production"
 gitlab_rails['db_username'] = "gitlab"
 gitlab_rails['db_password'] = "REDACTED PASSWORD"
 gitlab_rails['db_host'] = "ud-0010752.bREDACTEDb.com"
letsencrypt['enable'] = false
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = '/etc/gitlab/ssl/lws-git.cer'
nginx['ssl_certificate_key'] = 'REDACTED KEY'
pages_nginx['ssl_certificate'] = '/etc/gitlab/ssl/REDACTEDgitpagesdev.cer'
pages_nginx['ssl_certificate_key'] = 'REDACTED KEY'
nginx['custom_gitlab_server_config'] = "include /etc/gitlab/nginx_status.conf;\nlarge_client_header_buffers 4 32k;\nhttp2_max_field_size 32k;\nhttp2_max_header_size 64k;\n"
logging['svlogd_size'] = 209715200
logging['svlogd_num'] =  30
logging['svlogd_timeout'] =  86400
logging['svlogd_filter'] =  'gzip'
gitaly['enable'] = true
gitaly['configuration'] = {
  listen_addr: '0.0.0.0:8075',
  prometheus_listen_addr: '0.0.0.0:9236',
  auth: {
    token: 'REDACTED TOKEN',
  },
  storage: [
    {
      name: 'default',
      path: '/var/opt/gitlab/git-data/repositories',
    },
  ],
  hooks: {
    custom_hooks_dir: '/var/opt/gitlab/cbs-sdlc/cbs-new-scripts/git/hooks/gitLabCustomHooks',
  },
}
git_data_dirs({
  "default" => {
        "gitaly_address" => 'tcp://REDACTEDgitdev.bREDACTEDb.com:8075', 'path' => '/var/opt/gitlab/git-data',
        "gitaly_token" => 'REDACTED TOKEN'
  },
  "gitaly_cluster" => {
    "gitaly_address" => "tcp://ud006333:2305",
    "gitaly_token" => 'REDACTED TOKEN'
  }
})
registry_external_url 'https://REDACTEDgitdev.bREDACTEDb.com:5050'
registry_nginx['ssl_certificate'] = '/etc/gitlab/ssl/lws-git.cer'
registry_nginx['ssl_certificate_key'] = 'REDACTED KEY'
gitlab_rails['external_diffs_enabled'] = true
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = true
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'region' => '',
  'aws_access_key_id' => 'REDACTED KEY',
  'aws_secret_access_key' => 'REDACTED KEY',
  'endpoint' => 'https://vm028783.bREDACTEDb.com',
  'path_style' => true
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['packages']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'REDACTED KEY'
gitlab_rails['object_store']['objects']['pages']['bucket'] = 'REDACTED KEY'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "bridgeheads.bREDACTEDb.com"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = "REDACTED.uk"
gitlab_rails['gitlab_email_from'] = 'REDACTED EMAIL'
gitlab_rails['gitlab_email_reply_to'] = 'REDACTED EMAIL'
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_ssl'] = false
gitlab_rails['smtp_force_ssl'] = false
gitlab_rails['geo_node_name'] = 'REDACTED_GitLab_Geo'
postgresql['sql_replication_password'] = "REDACTED PASSWORD"
redis['enable'] = false
redis['master_name'] = 'REDACTED-gitlab'
redis['master_password'] = "REDACTED PASSWORD"
gitlab_rails['redis_password'] = "REDACTED PASSWORD"
gitlab_rails['redis_sentinels'] = [
  {'host' => 'ud-0010752.bREDACTEDb.com', 'port' => '26379'}
]
```

Edited Aug 05, 2024 by Collen