Migrate check-packages pipeline to download packages from Pulp

What does this MR do?

Migrate check-packages pipeline to download packages from Pulp

Change the Omnibus package checking pipeline to use pulp.gitlab.com instead of packages.gitlab.com.

• Add configurable package_server to install_package.sh
• Pass Pulp basic auth credentials (username/password) to install scripts
• Fetch PULP_PASSWORD_RO and PULP_USER_RO from https://dev.gitlab.org/gitlab/omnibus-gitlab/-/settings/ci_cd#js-cicd-variables-settings
• Set check-packages-functionality to use pre-release repo on Pulp

Ref: gitlab-org/build/team-tasks#162

Testing

Start a Docker container using the same image used by Omnibus, e.g. https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/34303624/viewer

docker run -it --rm registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/ubuntu_20.04_arm64:5.50.0 bash

Clone and checkout the branch

apt update && apt install git vim -y
git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git && cd omnibus-gitlab && git checkout pre-release-check-pulp 
bash scripts/ci/prepare_bundle.sh

In install_package.sh , manually change package_name_version_dist='gitlab-ee=18.9.0-ee.0' . On the pipeline, it is set by the commit tag.

Run

PULP_URL=pulp.gitlab.com PULP_USER=omnibus-downloader PULP_PASSWORD='password' package_type=deb package_manager=apt package_repository=pre-release bash scripts/ci/install_package.sh

All works well. GitLab is installed. We can see that pulp.gitlab.com is used:

Installing gitlab-ce using https://pulp.gitlab.com/install/repositories/gitlab/pre-release/script.deb.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    10  100    10    0     0     36      0 --:--:-- --:--:-- --:--:--    36
100  5287  100  5287    0     0   4569      0  0:00:01  0:00:01 --:--:--  4569
Hit:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Hit:5 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Hit:4 https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal focal InRelease
Reading package lists... Done
N: Usage of apt_auth.conf(5) should be preferred over embedding login information directly in the sources.list(5) entry for 'https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal'
Repository configured successfully.
Ready to install packages.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  gitlab-ee
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 1463 MB of archives.
After this operation, 4581 MB of additional disk space will be used.
Get:1 https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal focal/main arm64 gitlab-ee arm64 18.9.0-ee.0 [1463 MB]
Fetched 1463 MB in 1min 40s (14.6 MB/s)                                                                                                                      
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package gitlab-ee.
(Reading database ... 23046 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_18.9.0-ee.0_arm64.deb ...
Unpacking gitlab-ee (18.9.0-ee.0) ...

Progress: [ 20%] [###########################.............................................................................................................] 

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.