Admin > Gitaly servers reports external Gitaly as not up to date on 19.4 when the version string carries a -g<revision> suffix

Summary

On 19.4, Admin > Gitaly servers shows a red cross in the "Up to date" column for external Gitaly and Praefect storages that are running the correct, matching version. The cluster is healthy and fully upgraded; only the indicator is wrong.

The cause is a version string format introduced in 19.4 that Gitaly::Server#expected_version? does not recognise. Any 19.4 installation using the Linux package (Omnibus) for external Gitaly or Praefect is affected, so this is not specific to one customer's build pipeline.

Steps to reproduce

  1. Deploy GitLab 19.4.1 with external Gitaly behind Praefect, using the Linux package (for example the gitlab/gitlab-ee:19.4.1-ee.0 image) on the Gitaly and Praefect nodes.
  2. Go to Admin > Gitaly servers.
  3. The virtual storage row shows a red cross under "Up to date".

Confirm from a Rails node or the Toolbox pod:

puts "Expected: #{Gitlab::GitalyClient.expected_server_version}"
Gitaly::Server.all.each do |s|
  puts "#{s.storage}: #{s.server_version} -> up_to_date=#{s.up_to_date?}"
end

Current behaviour

Expected: 19.4.1
default:                19.4.1          -> up_to_date=true
gitaly-virtual-storage: 19.4.1-g7985a4e -> up_to_date=false

Both storages run 19.4.1. 7985a4e is the commit at the v19.4.1 tag in gitlab-org/gitaly, so the binary is the correct build. The chart-deployed default storage passes and the Linux package storage behind Praefect fails.

Expected behaviour

A Gitaly server running the expected version should report as up to date regardless of whether the build appended a Git revision to the version string.

Root cause

Two changes combine.

1. The version string gained a -g<revision> suffix in 19.4.

The version is injected at build time from git describe. gitaly!9173 (merged) added a fallback for builds where git describe finds no reachable tag: it falls back to the VERSION file and appends the short revision. The MR documents the intended result as 19.2.0-rc1 becoming 19.2.0-rc1-gcd716d3.

That fallback is reached in the Linux package build because, as of 19.4, config/software/gitaly.rb fetches Gitaly with a shallow clone:

source git: version.remote, depth: 1

On 19.2.7 the same line had no depth option. A shallow clone has no reachable tag, so git describe produces nothing and the fallback appends the revision. CNG is unaffected because its Dockerfile clones with --filter=tree:0, a partial clone that retains tags.

2. expected_version? accepts neither the exact match nor the pattern.

lib/gitaly/server.rb:

SHA_VERSION_REGEX = /\A\d+\.\d+\.\d+-\d+-g([a-f0-9]{8})\z/

def expected_version?
  server_version == Gitlab::GitalyClient.expected_server_version || matches_sha?
end

19.4.1-g7985a4e fails the exact comparison against 19.4.1. It also fails SHA_VERSION_REGEX, which requires a commit count (-\d+-) and an 8 character revision. The new format has no commit count and a 7 character revision.

Loosening the regex alone is not sufficient

Worth flagging, because it is the non-obvious part. matches_sha? compares the captured revision against the expected version:

def matches_sha?
  match = server_version.match(SHA_VERSION_REGEX)
  return false unless match

  Gitlab::GitalyClient.expected_server_version.start_with?(match[1])
end

That branch assumes expected_server_version is itself a SHA, which holds on auto-deploy but not on a tagged release, where GITALY_SERVER_VERSION is 19.4.1. So "19.4.1".start_with?("7985a4e") is false, and widening the pattern to capture the revision would still return false. The comparison needs to use the semantic version portion of the string, not the revision.

Proposed fix

Treat X.Y.Z-g<revision> as a version match on its X.Y.Z prefix, leaving the existing exact match and auto-deploy SHA path untouched:

VERSION_WITH_REVISION_REGEX = /\A(\d+\.\d+\.\d+)-g[a-f0-9]{5,40}\z/

def expected_version?
  server_version == Gitlab::GitalyClient.expected_server_version ||
    matches_sha? ||
    matches_version_with_revision?
end

private

def matches_version_with_revision?
  match = server_version.match(VERSION_WITH_REVISION_REGEX)
  return false unless match

  match[1] == Gitlab::GitalyClient.expected_server_version
end

Offered as a starting point rather than a finished patch; the Gitaly group will know whether pre-release strings such as 19.2.0-rc1-gcd716d3 should be handled in the same place.

Note that lib/tasks/gitlab/gitaly.rake has a separate, looser check in warn_gitaly_out_of_date! using /g([a-f0-9]{5,40})\z/. It matches the new string, then hits the same start_with? comparison and prints a spurious does not exactly match repository version warning. It is only reachable from gitlab:gitaly:install, so it affects source installs and development rather than deployed instances, but it is the same underlying assumption and may be worth fixing together.

Do not fix this in Omnibus

The shallow clone was introduced deliberately to reduce Git egress from the build pipelines (gitlab-org/omnibus-gitlab#10035, a cost optimisation). Reverting it would undo that work. The version string it produces is correct and more precise than before, since it identifies the exact commit. The defect is that Rails rejects a valid format.

Impact

Cosmetic, and verified as such rather than assumed. up_to_date? has a single consumer in the codebase, app/views/admin/gitaly_servers/index.html.haml. It does not gate Git operations, replication, repository access, health and readiness checks or upgrades. Gitaly and Praefect do not parse their own version string, using it only for startup logs, --version output and a User-Agent header.

The practical cost is operational rather than functional:

  • Administrators cannot use this page to verify a Gitaly upgrade, which is the purpose documented at doc/administration/gitaly/troubleshooting.md#check-versions-when-using-standalone-gitaly-servers. That documentation tells users to "confirm all Gitaly servers indicate that they are up to date", which is now unachievable on an affected install.
  • Upgrade runbooks that include this check will fail against a healthy cluster, which invites a rollback of a successful upgrade.
  • In the originating ticket the customer paused a planned production upgrade while the indicator was investigated.

No workaround exists. The version is compiled into the binary, so no configuration setting changes what it reports.

Versions affected

  • Affected: 19.4.x installations where Gitaly or Praefect come from the Linux package, including the official Docker images.
  • Not affected: 19.2.7 and earlier, which used a full clone and produced a bare version.
  • Not affected: CNG and Helm chart deployments, which use a partial clone that retains tags.

A single install can show both states at once when storages come from different build methods, as in the output above.

Customer impact

Reported via a Premium self-managed ticket. Upgrade 19.2.7 to 19.4.1, Helm chart for the application with external Gitaly Cluster on GCP, three Gitaly nodes behind Praefect using gitlab/gitlab-ee:19.4.1-ee.0. Cluster health confirmed by praefect check, all four checks passed, three nodes healthy and consistent, no unavailable repositories. Production upgrade was paused pending an answer.

Edited by 🤖 GitLab Bot 🤖