Throw better error when docker commands fail to pull

Jay McCurerequested to merge
jmc-735 into master

What does this MR do and why?

Resolves #735 (closed)

In #735 (closed) we saw that when the GITLAB_QA_DEV_ACCESS_TOKEN gitlab-qa failed when retrying the tests, giving a misleading error message.

With this change, we don't retry the tests when pulling the gitlab-qa image fails.

Also when logging into Docker registries or pulling the gitlab-qa image fails, the code now raises more descriptive error messages.

Also added unit tests.

How to set up and validate locally

For GITLAB_QA_ACCESS_TOKEN below, use api-pat-gitlab-qa from 1Password entry: gitlab-qa - staging.gitlab.com - DO NOT ENABLE 2FA OR JOIN gitlab-org GROUP (non-admin) - ONLY FOR USE WITH QA AUTOMATION

From master:

gem build gitlab-qa.gemspec && gem install ./gitlab-qa-15.5.0.gem
GITLAB_QA_DEV_ACCESS_TOKEN=foo QA_RETRY_FAILED_SPECS=true QA_LOG_LEVEL=debug GITLAB_QA_ACCESS_TOKEN=<1Password> gitlab-qa Test::Instance::Staging  --address https://staging.gitlab.com --  "--tag smoke --tag ~orchestrated --format html"

Should give misleading error:

[May 19 2025 14:34:54 AEST (Gitlab QA)] WARN  -- Initial test run failed, attempting to retry failed specs in new process!
/Users/jmccure/.local/share/mise/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/gitlab-qa-15.5.0/lib/gitlab/qa/component/specs.rb:222:in `join': no implicit conversion of nil into String (TypeError)

          File.join(Runtime::Env.host_artifacts_dir, name)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now from this branch do the same:

gem build gitlab-qa.gemspec && gem install ./gitlab-qa-15.5.0.gem
GITLAB_QA_DEV_ACCESS_TOKEN=foo QA_RETRY_FAILED_SPECS=true QA_LOG_LEVEL=debug GITLAB_QA_ACCESS_TOKEN=<1Password> gitlab-qa Test::Instance::Staging  --address https://staging.gitlab.com --  "--tag smoke --tag ~orchestrated --format html"

Should give a more useful error and the tests will not be retried:

/Users/jmccure/.local/share/mise/installs/ruby/3.3.7/lib/ruby/gems/3.3.0/gems/gitlab-qa-15.5.0/lib/gitlab/qa/component/specs.rb:165:in `rescue in docker_pull_qa_image_if_needed': Failed to login to Docker registry: Command `docker login --username "gitlab-qa-bot" --password "*****" dev.gitlab.org:5005` failed! ✘ (RuntimeError)

Note: i have run the omnibus pipeline for this MR: https://gitlab.com/gitlab-org/gitlab-qa/-/pipelines/1824048296 (failures are test failures and are unrelated to this change)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jay McCure

Merge request reports