Upgrade to 18.2.0-ce: Error "Cookbook gitlab-ee not found" on a gitlab-ce instance?

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Maybe related to #556374 (closed)?

Summary

Upgrade from 18.1.2-ce to 18.2.0-ce failed. Our GitLab instance is down.

Note that this server was initially a gitlab-ee, but we had to move to gitlab-ce 5 years ago. Since then, GitLab CE upgrades have been working flawlessly through Unattended-Upgrades. This is the first time it fails.

Steps to reproduce

Since upgrade, now running sudo gitlab-ctl reconfigure fails:

See output 
[2025-07-18T09:48:38+02:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
Cinc Client, version 18.3.0
Patents: https://www.chef.io/patents
Infra Phase starting
[2025-07-18T09:48:38+02:00] INFO: *** Cinc Client 18.3.0 ***
[2025-07-18T09:48:38+02:00] INFO: Platform: x86_64-linux
[2025-07-18T09:48:38+02:00] INFO: Cinc-client pid: 57020
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/encoder.rb:42: warning: undefining the allocator of T_DATA class FFI_Yajl::Ext::Encoder::YajlGen
[2025-07-18T09:48:39+02:00] INFO: Setting the run_list to ["recipe[gitlab-base]"] from CLI options
[2025-07-18T09:48:39+02:00] INFO: Run List is [recipe[gitlab-base]]
[2025-07-18T09:48:39+02:00] INFO: Run List expands to [gitlab-base]
[2025-07-18T09:48:39+02:00] INFO: Starting Cinc Client Run for gitlab1.example.com
[2025-07-18T09:48:39+02:00] INFO: Running start handlers
[2025-07-18T09:48:39+02:00] INFO: Start handlers complete.
Resolving cookbooks for run list: ["gitlab-base"]
[2025-07-18T09:48:41+02:00] INFO: Loading cookbooks [gitlab-base@0.1.0, package@0.1.0, gitlab@0.0.1, runit@5.1.7, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, regist
ry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.6, crond@0.1.0]
Synchronizing cookbooks:
  - gitlab-base (0.1.0)
  - package (0.1.0)
  - gitlab (0.0.1)
  - runit (5.1.7)
  - logrotate (0.1.0)
  - redis (0.1.0)
  - postgresql (0.1.0)
  - mattermost (0.1.0)
  - registry (0.1.0)
  - praefect (0.1.0)
  - monitoring (0.1.0)
  - letsencrypt (0.1.0)
  - consul (0.1.0)
  - gitlab-kas (0.1.0)
  - gitlab-pages (0.1.0)
  - gitaly (0.1.0)
  - nginx (0.1.0)
  - acme (4.1.6)
  - crond (0.1.0)
Installing cookbook gem dependencies:
Compiling cookbooks...
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:2: warning: already initialized constant SELinuxDistroHelper::REDHAT_RELEASE_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:2: warning: previous definition of REDHAT_RELEASE_FILE was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:3: warning: already initialized constant SELinuxDistroHelper::OS_RELEASE_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:3: warning: previous definition of OS_RELEASE_FILE was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:4: warning: already initialized constant SecretsHelper::SECRETS_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:4: warning: previous definition of SECRETS_FILE was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:5: warning: already initialized constant SecretsHelper::SECRETS_FILE_CHEF_ATTR
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:5: warning: previous definition of SECRETS_FILE_CHEF_ATTR was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:6: warning: already initialized constant SecretsHelper::SKIP_GENERATE_SECRETS_CHEF_ATTR
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:6: warning: previous definition of SKIP_GENERATE_SECRETS_CHEF_ATTR was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/gitlab_cluster.rb:16: warning: already initialized constant GitlabCluster::CONFIG_PATH
/opt/gitlab/embedded/cookbooks/package/libraries/gitlab_cluster.rb:16: warning: previous definition of CONFIG_PATH was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/gitlab_cluster.rb:17: warning: already initialized constant GitlabCluster::JSON_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/gitlab_cluster.rb:17: warning: previous definition of JSON_FILE was here
Loading Cinc Auditor profile files:
Loading Cinc Auditor input files:
Loading Cinc Auditor waiver files:
[2025-07-18T09:48:42+02:00] WARN: MissingCookbookDependency:
Recipe `gitlab-ee::default` is not in the run_list, and cookbook 'gitlab-ee'
is not a dependency of any cookbook in the run_list. To load this recipe,
first add a dependency of the cookbook 'gitlab-ee' into the metadata
          of the cookbook which depends on 'gitlab-ee'.


================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb
================================================================================

Chef::Exceptions::CookbookNotFound
----------------------------------
Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata

Cookbook Trace: (most recent call first)
----------------------------------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb:4:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb:

  1:  if Dir.exist?(File.absolute_path(File.join(__dir__, "../../gitlab-jh")))
  2:    include_recipe 'gitlab-jh::default'
  3:  elsif Dir.exist?(File.absolute_path(File.join(__dir__, "../../gitlab-ee")))
  4>>   include_recipe 'gitlab-ee::default'
  5:  elsif Dir.exist?(File.absolute_path(File.join(__dir__, "../../gitlab")))
  6:    include_recipe 'gitlab::default'
  7:  end
  8:

System Info:
------------
chef_version=18.3.0
platform=ubuntu
platform_version=24.04
ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client


Running handlers:
[2025-07-18T09:48:42+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata

Running handlers complete
[2025-07-18T09:48:42+02:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 04 seconds
[2025-07-18T09:48:42+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2025-07-18T09:48:42+02:00] FATAL: ---------------------------------------------------------------------------------------
[2025-07-18T09:48:42+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2025-07-18T09:48:42+02:00] FATAL: ---------------------------------------------------------------------------------------
[2025-07-18T09:48:42+02:00] FATAL: Chef::Exceptions::CookbookNotFound: Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata

gitlab-ctl check-config is failing too:

/opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:93:in `applicable_deprecations': undefined method `empty?' for nil:NilClass (NoMethodError)

        incoming_version = next_major_version if incoming_version.empty?
                                                                 ^^^^^^^
        from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:111:in `check_config'
        from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:51:in `block in load_file'
        from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:197:in `block in add_command'
        from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:747:in `run'
        from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omnibus-ctl-0.6.12/bin/omnibus-ctl:31:in `<top (required)>'
        from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
        from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

System information
System:         Ubuntu 24.04
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.6.9
Bundler Version:2.6.5
Rake Version:   13.0.6
Redis Version:  7.2.9
Sidekiq Version:7.3.9
Go Version:     unknown

GitLab information
Version:        18.2.0
Revision:       95351f08aae
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     16.8
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        14.43.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      18.2.0
- default Git Version:  2.50.1.gl1

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.43.0 ? ... OK (14.43.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check
Internal API available: FAILED - Internal API unreachable
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... Checking gitlab@gitlab.example.com
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 16 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... no
Try fixing it:
sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
Please fix the error above and rerun the checks.
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
3/1 ... yes
16/6 ... yes
[...]
41/89 ... yes
130/90 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.2.5)
Git user has default SSH configuration? ... yes
Active users: ... 24
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by 🤖 GitLab Bot 🤖