## Summary On 19.4, **Admin > Gitaly servers** shows a red cross in the "Up to date" column for external Gitaly and Praefect storages that are running the correct, matching version. The cluster is healthy and fully upgraded; only the indicator is wrong. The cause is a version string format introduced in 19.4 that `Gitaly::Server#expected_version?` does not recognise. Any 19.4 installation using the Linux package (Omnibus) for external Gitaly or Praefect is affected, so this is not specific to one customer's build pipeline. ## Steps to reproduce 1. Deploy GitLab 19.4.1 with external Gitaly behind Praefect, using the Linux package (for example the `gitlab/gitlab-ee:19.4.1-ee.0` image) on the Gitaly and Praefect nodes. 1. Go to **Admin > Gitaly servers**. 1. The virtual storage row shows a red cross under "Up to date". Confirm from a Rails node or the Toolbox pod: ```ruby puts "Expected: #{Gitlab::GitalyClient.expected_server_version}" Gitaly::Server.all.each do |s| puts "#{s.storage}: #{s.server_version} -> up_to_date=#{s.up_to_date?}" end ``` ## Current behaviour ``` Expected: 19.4.1 default: 19.4.1 -> up_to_date=true gitaly-virtual-storage: 19.4.1-g7985a4e -> up_to_date=false ``` Both storages run 19.4.1. `7985a4e` is the commit at the `v19.4.1` tag in `gitlab-org/gitaly`, so the binary is the correct build. The chart-deployed `default` storage passes and the Linux package storage behind Praefect fails. ## Expected behaviour A Gitaly server running the expected version should report as up to date regardless of whether the build appended a Git revision to the version string. ## Root cause Two changes combine. **1. The version string gained a `-g<revision>` suffix in 19.4.** The version is injected at build time from `git describe`. gitlab-org/gitaly!9173 added a fallback for builds where `git describe` finds no reachable tag: it falls back to the `VERSION` file and appends the short revision. The MR documents the intended result as `19.2.0-rc1` becoming `19.2.0-rc1-gcd716d3`. That fallback is reached in the Linux package build because, as of 19.4, `config/software/gitaly.rb` fetches Gitaly with a shallow clone: ```ruby source git: version.remote, depth: 1 ``` On 19.2.7 the same line had no `depth` option. A shallow clone has no reachable tag, so `git describe` produces nothing and the fallback appends the revision. CNG is unaffected because its Dockerfile clones with `--filter=tree:0`, a partial clone that retains tags. **2. `expected_version?` accepts neither the exact match nor the pattern.** `lib/gitaly/server.rb`: ```ruby SHA_VERSION_REGEX = /\A\d+\.\d+\.\d+-\d+-g([a-f0-9]{8})\z/ def expected_version? server_version == Gitlab::GitalyClient.expected_server_version || matches_sha? end ``` `19.4.1-g7985a4e` fails the exact comparison against `19.4.1`. It also fails `SHA_VERSION_REGEX`, which requires a commit count (`-\d+-`) and an 8 character revision. The new format has no commit count and a 7 character revision. ### Loosening the regex alone is not sufficient Worth flagging, because it is the non-obvious part. `matches_sha?` compares the captured revision against the expected version: ```ruby def matches_sha? match = server_version.match(SHA_VERSION_REGEX) return false unless match Gitlab::GitalyClient.expected_server_version.start_with?(match[1]) end ``` That branch assumes `expected_server_version` is itself a SHA, which holds on auto-deploy but not on a tagged release, where `GITALY_SERVER_VERSION` is `19.4.1`. So `"19.4.1".start_with?("7985a4e")` is false, and widening the pattern to capture the revision would still return false. The comparison needs to use the semantic version portion of the string, not the revision. ## Proposed fix Treat `X.Y.Z-g<revision>` as a version match on its `X.Y.Z` prefix, leaving the existing exact match and auto-deploy SHA path untouched: ```ruby VERSION_WITH_REVISION_REGEX = /\A(\d+\.\d+\.\d+)-g[a-f0-9]{5,40}\z/ def expected_version? server_version == Gitlab::GitalyClient.expected_server_version || matches_sha? || matches_version_with_revision? end private def matches_version_with_revision? match = server_version.match(VERSION_WITH_REVISION_REGEX) return false unless match match[1] == Gitlab::GitalyClient.expected_server_version end ``` Offered as a starting point rather than a finished patch; the Gitaly group will know whether pre-release strings such as `19.2.0-rc1-gcd716d3` should be handled in the same place. Note that `lib/tasks/gitlab/gitaly.rake` has a separate, looser check in `warn_gitaly_out_of_date!` using `/g([a-f0-9]{5,40})\z/`. It matches the new string, then hits the same `start_with?` comparison and prints a spurious `does not exactly match repository version` warning. It is only reachable from `gitlab:gitaly:install`, so it affects source installs and development rather than deployed instances, but it is the same underlying assumption and may be worth fixing together. ### Do not fix this in Omnibus The shallow clone was introduced deliberately to reduce Git egress from the build pipelines (gitlab-org/omnibus-gitlab#10035, a cost optimisation). Reverting it would undo that work. The version string it produces is correct and more precise than before, since it identifies the exact commit. The defect is that Rails rejects a valid format. ## Impact Cosmetic, and verified as such rather than assumed. `up_to_date?` has a single consumer in the codebase, `app/views/admin/gitaly_servers/index.html.haml`. It does not gate Git operations, replication, repository access, health and readiness checks or upgrades. Gitaly and Praefect do not parse their own version string, using it only for startup logs, `--version` output and a User-Agent header. The practical cost is operational rather than functional: - Administrators cannot use this page to verify a Gitaly upgrade, which is the purpose documented at `doc/administration/gitaly/troubleshooting.md#check-versions-when-using-standalone-gitaly-servers`. That documentation tells users to "confirm all Gitaly servers indicate that they are up to date", which is now unachievable on an affected install. - Upgrade runbooks that include this check will fail against a healthy cluster, which invites a rollback of a successful upgrade. - In the originating ticket the customer paused a planned production upgrade while the indicator was investigated. No workaround exists. The version is compiled into the binary, so no configuration setting changes what it reports. ## Versions affected - **Affected:** 19.4.x installations where Gitaly or Praefect come from the Linux package, including the official Docker images. - **Not affected:** 19.2.7 and earlier, which used a full clone and produced a bare version. - **Not affected:** CNG and Helm chart deployments, which use a partial clone that retains tags. A single install can show both states at once when storages come from different build methods, as in the output above. ## Customer impact Reported via a Premium self-managed ticket. Upgrade 19.2.7 to 19.4.1, Helm chart for the application with external Gitaly Cluster on GCP, three Gitaly nodes behind Praefect using `gitlab/gitlab-ee:19.4.1-ee.0`. Cluster health confirmed by `praefect check`, all four checks passed, three nodes healthy and consistent, no unavailable repositories. Production upgrade was paused pending an answer.