Add upgrade test coverage between internal releases

Nailia Iskhakovarequested to merge
add-internal-patch-upgrade-path-scenario-option into master

What does this MR do and why?

This MR adds a new internal_patch upgrade path option to be used with UpdateFromPrevious scenario. This allows to use internal pre-release builds for testing upgrades - https://gitlab.com/gitlab-com/gl-infra/software-delivery/framework/software-delivery-framework-issue-tracker/-/issues/29+

Internal releases has the format gitlab-com/gl-infra/delivery#20812 (comment 2394813225) dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0 and are stored in dev registry which requires authentication. Also searching the image requires pulling from the registry to check if internal release is available, as API search is not available for registry and internal releases don't have tags, see https://gitlab.com/gitlab-com/gl-infra/software-delivery/framework/software-delivery-framework-issue-tracker/-/issues/29#note_2395173347

internal_patch is separate from patch as in backport MRs, upgrade paths for internal patch -> backport_MR and patch -> backport_MR need to be covered both, as first one is Dedicated specific and second one is more wide and also covers general self-managed customers.

How to set up and validate locally

  1. Run the command locally - note that test tag of 99 was already removed gitlab-com/gl-infra/delivery#20812 (comment 2394813225) and so below command tests that no internal release is available for testing
CHROME_DISABLE_DEV_SHM=true GITLAB_QA_DEV_ACCESS_TOKEN=XXX bundle exec gitlab-qa Test::Omnibus::UpdateFromPrevious gitlab/gitlab-ee:17.7.7-ee.0 17.7.7-ee internal_patch  -- --tag health_check

...

[Mar 17 2025 17:55:40 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 13 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:55:40 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-qa-bot" --password "*****" dev.gitlab.org:5005`
......
[Mar 17 2025 17:55:41 UTC (Gitlab QA)] INFO  -- Start searching for the latest released internal image for gitlab version: 17.7.7...
....

[Mar 17 2025 17:55:43 UTC (Gitlab QA)] ERROR -- Shell command output:
Error response from daemon: manifest for dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal0-0 not found: manifest unknown: manifest unknown
[Mar 17 2025 17:55:43 UTC (Gitlab QA)] INFO  -- x - Image not found: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal0-0,
 Command `docker pull -q dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal0-0` failed! ✘
[Mar 17 2025 17:55:43 UTC (Gitlab QA)] WARN  -- No internal image found for GitLab version 17.7.7
  1. With dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0 available locally we see the following (with tweaked search using 100.downto(98)) => Performing gitlab update: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0 => gitlab/gitlab-ee:17.7.7-ee.0 (17.7.7-ee) ✅
Click to expand 
CHROME_DISABLE_DEV_SHM=true GITLAB_QA_DEV_ACCESS_TOKEN=xxxx bundle exec gitlab-qa Test::Omnibus::UpdateFromPrevious gitlab/gitlab-ee:17.7.7-ee.0 17.7.7-ee internal_patch  -- --tag health_check
[Mar 17 2025 17:58:35 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 1 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:35 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 2 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:35 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 3 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:36 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 4 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:36 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 5 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:36 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 6 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:36 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 7 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:37 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 8 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:37 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 9 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:37 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 10 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:37 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 11 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:38 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 12 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:38 UTC (Gitlab QA)] INFO  -- Fetching Docker tags page 13 from 'gitlab/gitlab-ee' registry
[Mar 17 2025 17:58:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker login --username "gitlab-qa-bot" --password "*****" dev.gitlab.org:5005`
......
[Mar 17 2025 17:58:38 UTC (Gitlab QA)] INFO  -- Start searching for the latest released internal image for gitlab version: 17.7.7...
[Mar 17 2025 17:58:38 UTC (Gitlab QA)] INFO  -- Checking for image: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal100-0
[Mar 17 2025 17:58:38 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal100-0`
.
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] ERROR -- Shell command output:
Error response from daemon: manifest for dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal100-0 not found: manifest unknown: manifest unknown
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- x - Image not found: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal100-0,
 Command `docker pull -q dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal100-0` failed! ✘
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Checking for image: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker pull -q dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0`
.
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Found image: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Performing gitlab update: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0 => gitlab/gitlab-ee:17.7.7-ee.0 (17.7.7-ee)
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Running the first release in upgrade path: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Deploying release: dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee:17.7.7-internal99-0
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^alpine-7af23594$`
[Mar 17 2025 17:58:39 UTC (Gitlab QA)] INFO  -- Shell command: `docker run -d --name alpine-7af23594 --volume authority:/etc/gitlab/trusted-certs:Z --volume gitlab-ssl:/etc/gitlab/ssl:Z alpine/curl:latest tail -f /dev/null`

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports