Generate a public version manifest for Omnibus GitLab

What does this MR do?

Create a version manifest for Omnibus Gitlab to publicly publish alongside software licenses. The version manifest should point to publicly available sources.

Related issues

• Closes #6909 (closed)
• Dependent release-tools!2125 (merged)

Design

We follow the pattern used by the license_upload pipeline in the license-upload stage.

• Create a new manifest-upload job in metrics stage of dev.gitlab.org.yaml.
• The manifest-upload job runs bundle exec rake manifest:upload.
• The manifest:upload task regenerates the versions-manifest.json file while setting ALTERNATIVE_SOURCES=true to get public references to repositories (gitlab.com not dev.gitlab.org). This file is then copied to the LICENSE_S3_BUCKET AWS S3 bucket.
• Modify the pages CI job in gitlab.com.yml to add the bundle exec rake manifest:generate_pages task.
• The manifest:generate_pages task will mimics the same code used used for license uploading including using a template to create the HTML page used to reference the manifests for each published version.
• These jobs are only run during releases so the job will initially allow_failure have set. This prevents breaking release builds until we confirm that it works correctly.

Future considerations

• Remove the allow_failure setting once confirmed that code runs without error.
• The license-upload and manifest-upload jobs should probably be combined into one job to provide a more DRY solution.

Test plan

• Create AWS S3 bucket in the eu-west-1 region with a descriptive name (this example will use version-manifests. DO NOT use the existing LICENSE_S3_BUCKET bucket. Enable access with an AWS id and key.
• It is helpful to access this bucket using a s3fs mount. The rest of this test plan assumes you have done so. Alternatively, you can browse/modify the contents using the AWS console. To mount the bucket use:
  • Install s3fs
  • Mount the file system

    AWS_ACCESS_KEY_ID=<aws_access_key_id> AWS_SECRET_ACCESS_KEY=<aws_secret_access_key> s3fs version-manifests /mnt/tmp

• Start docker build container (see https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/build/build_package.md for details)

   export TGT_OS=debian_11
   export BUILDER_IMAGE_REVISION=4.1.0
   docker run --rm --name omnibus-builder-${BUILDER_IMAGE_REVISION} -v ~/<omnibus-gitlab-src_dir>:/omnibus-gitlab -v <assets_dir>:/gitlab-assets -it registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/${TGT_OS}:${BUILDER_IMAGE_REVISION} bash

• Prepare build environment

  export ASSET_PATH=/gitlab-assets
  cd /omnibus-gitlab
  bundle install
  bundle binstubs --all
  export LICENSE_S3_BUCKET=version-manifests
  export LICENSE_AWS_ACCESS_KEY_ID=<aws access_key_id>
  export LICENSE_AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>

• Create license manifests for both EE and CE editions for three different versions of the product. The first version will just use the version provided in the current branch. We change the version for test purposes by editing lib/gitlab/build/info.rb

  diff --git a/lib/gitlab/build/info.rb b/lib/gitlab/build/info.rb
  index ac3f7e077..9f2b94f84 100644
  --- a/lib/gitlab/build/info.rb
  +++ b/lib/gitlab/build/info.rb
  @@ -54,6 +54,7 @@ module Build
             ver_tag += ".fips" if Build::Check.use_system_ssl?
             [ver_tag, Gitlab::Util.get_env('CI_PIPELINE_ID'), commit_sha].compact.join('.')
           end
  +        '15.7.1+123def'
         end
   
         def branch_name

  Note that the +123def is a fake commit sha. It is required.
• Use the following command to create an EE and CE version manifest for each version.

   for ee in false true; do rm -rf rm -rf version-manifests /var/cache/omnibus/manifests; ee=${ee} bundle exec rake manifest:upload; done

  • Run using current version X.Y.0 (do not modify info.rb).
  • Run using with a incremented patch level (set version in info.rb to X.Y.1+123def).
  • Run using with a incremented minor number (set version in info.rb to X.Y+1.0+123def).
• Verify that the S3 bucket has the correct contents.
  • Directory structure:

  /mnt/tmp
  └── gitlab-manifests
      ├── gitlab-ce
      │   ├── 15.7
      │   │   ├── 15.7.0-ce.version-manifest.json
      │   │   └── 15.7.1-ce.version-manifest.json
      │   └── 15.8
      │       └── 15.8.0-ce.version-manifest.json
      └── gitlab-ee
          ├── 15.7
          │   ├── 15.7.0-ee.version-manifest.json
          │   └── 15.7.1-ee.version-manifest.json
          └── 15.8
              └── 15.8.0-ee.version-manifest.json
  
  7 directories, 6 files

  • Make sure all version-manifest.json files do not have URL to dev.gitlab.org
  • Make sure all ee-version-manifest.json files include references to proprietary repos, e.g. gitlab-ctl-ee.
  • Make sure all ce-version-manifest.json files DO NOT include references to proprietary repos, e.g. gitlab-ctl-ee.
• Run manifest:generate_pages to generate the HTML page and check output.

  bundle exec rake manifest:generate_pages

  Resulting manfest.html file should be in version-manifests/manifest.html. Check contents to see if it is generated correctly.

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened

Closes #6909 (closed)