### Summary When attempting to upgrade GitLab Omnibus from version 18.11.7 to 19.1.2, the installation via apt install fails during the pre-installation check with an error stating that bundled Mattermost has been removed. However, Mattermost has never been enabled or used on this instance. All `mattermost[...]` configuration keys in `/etc/gitlab/gitlab.rb` are completely commented out or absent. A similar issue has been reported for Docker-based installations in the GitLab Forum: [Docker upgrade 18.11.3 -> 19.0.0 fails](https://forum.gitlab.com/t/docker-upgrade-18-11-3-19-0-0-fails-container-exits-30s-after-start-dependency-failed-to-start-container-gitlab-is-unhealthy/134005) While Docker users can work around this by manually altering the environment or config files within the container, the Omnibus package completely blocks the apt upgrade pipeline because the package pre-flight check detects stale Mattermost attributes cached in the system state. ### Steps to reproduce `sudo apt install gitlab-ee=19.1.2-ee.0` ### What is the current *bug* behavior? Even if Mattermost is disabled, earlier GitLab 18.x Omnibus installations automatically populate two places with default Mattermost references: The `/etc/gitlab/gitlab-secrets.json` file (which generates a `"mattermost": {...}` block with secrets by default). The Chef node state cache located at `/opt/gitlab/embedded/nodes/<hostname>.json.` During apt install, the preinst script evaluates these internal configuration caches. Because they contain the Mattermost structure, the upgrade aborts, creating a false positive block for instances that never actually deployed Mattermost. ### Relevant logs <details> <summary> Relevant logs </summary> <pre> * mattermost has been deprecated since 19.0 and was removed in 19.0. Bundled Mattermost has been removed from the Linux package in 19.0; `mattermost[...]` keys are no longer supported. Deploy Mattermost separately and point GitLab at it with `gitlab_rails['mattermost_host']`. See https://docs.gitlab.com/integration/mattermost/#running-gitlab-mattermost-on-its-own-server for setup instructions. Deprecations found. Please correct them and try again. dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_19.1.2-ee.0_amd64.deb (--unpack): new gitlab-ee package pre-installation script subprocess returned error exit status 1 </pre> </details> ### Workaround #### Linux package To successfully bypass this blocker and complete the upgrade, the following steps were required: - Remove Mattermost from secrets: Manually edit `/etc/gitlab/gitlab-secrets.json` and delete the entire `"mattermost": { ... }` block. - Prevent automatic reconfigure during package install: ``` sudo touch /etc/gitlab/skip-auto-reconfigure ``` - Clear the Chef node state cache: Delete the existing configuration JSON file inside the embedded nodes directory to prevent the installer from reading the deprecated cached state: ``` sudo rm /opt/gitlab/embedded/nodes/*.json ``` Note: This forces the installer to output `JSON file with existing configuration not found inside /opt/gitlab/embedded/nodes. Skipping config check.`, allowing the package upgrade to safely proceed. - Run the upgrade: ``` sudo apt install gitlab-ee=19.1.2-ee.0 ``` - Finalize: Remove the skip file and trigger the reconfigure manually to regenerate a clean state for GitLab 19.x: ``` sudo rm /etc/gitlab/skip-auto-reconfigure sudo gitlab-ctl reconfigure ``` #### Docker To successfully bypass this blocker and complete the upgrade, the following steps were required: - Remove Mattermost from secrets when on image 18.11.7: Manually edit `/etc/gitlab/gitlab-secrets.json` and delete the entire `"mattermost": { ... }` block. - Do not reconfigure after editing the secrets, otherwise `gitlab-omnibus` would [re-add mattermost ](https://docs.gitlab.com/omnibus/development/reconfigure_in_detail/#handling-of-secrets)block back. - Proceed upgrade process to 19.x https://docs.gitlab.com/update/docker/ ### Root cause GitLab generated default Mattermost salts unconditionally on every reconfigure. [`GitlabMattermost.parse_secrets`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/18-11-stable/files/gitlab-cookbooks/gitlab/libraries/gitlab_mattermost.rb#L31-37) sets `email_invite_salt`, `file_public_link_salt`, `sql_at_rest_encrypt_key`, `gitlab_id` and `gitlab_secret` via `||= SecretsHelper.generate_hex(...)` with **no `enable` guard**. It is invoked for every install via [`SettingsDSL#generate_secrets`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/18-11-stable/files/gitlab-cookbooks/package/libraries/settings_dsl.rb#L165-185). Those values are then [written to `gitlab-secrets.json`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/18-11-stable/files/gitlab-cookbooks/package/libraries/helpers/secrets_helper.rb#L120-124) and persisted into the node cache on reconfigure. They remain even after every `mattermost[...]` key is removed from `gitlab.rb`, so the check sees leftover secret state, treats it as active removed configuration, and aborts the upgrade with a false positive. ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-============================================================ un gitlab-ce <none> <none> (no description available) ii gitlab-ee 19.1.2-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis) un gitlab-fips <none> <none> (no description available) </pre> </details> ### Environment details * Operating System: `Ubuntu 22.04 LTS` * Installation Target: * VM: AWS * Installation Type: * Upgrade from version `18.11.7` * This is a single node installation * Resources * vCPU: `4` * Memory total: `16Gi`