### Problem to solve While working on [Build docker images on a regular schedule](https://gitlab.com/gitlab-org/gitlab/-/issues/338826), there was a [discussion](https://gitlab.com/gitlab-org/security-products/ci-templates/-/merge_requests/255#note_702235232) about whether we should fetch the most recent version of the analyzer from the `git tag` or from the `CHANGELOG.md` file. A concern was raised that if a developer forgets to tag and release an analyzer, the most recent version in the `CHANGELOG.md` file would not match the most recent `git tag`, since a tag wasn't created. It was decided that we could address this issue, but it was out of scope, so that's why this new issue is being created to track it. ### Proposal The scheduled job for automatically building Docker images (see [gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/pipeline_schedules), [retire.js](https://gitlab.com/gitlab-org/security-products/analyzers/retire.js/-/pipeline_schedules), [bundler-audit](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit/-/pipeline_schedules), etc) currently does the following: 1. [Fetches the most recent version of the analyzer using `git tag -l`](https://gitlab.com/gitlab-org/security-products/ci-templates/-/blob/4ee6b59baafa3666753836a75e51f279a6d42dca/includes-dev/docker.yml#L72) 1. Rebuilds the analyzer with the version value fetched in step `1.` We should update this process to the following: 1. Fetch the most recent version of the analyzer and the description of the change from the `CHANGELOG.md` file. 1. Check to see if a tag already exists with the given version value fetched in step `1.`: - If such a tag exists, then rebuild a new analyzer Docker image with that tag - If a tag does not exist, tag a new analyzer Docker image with that tag and provide a tag description which matches the description for the change from the `CHANGELOG.md` file ### Implementation Plan Create an MR with the following changes, and notify the Static Analysis team about the new `CREATE_GIT_TAG` variable in case they also want to make use of it: 1. Create a new stage: `tag` (this contains a job for creating the git tag from the most recent changelog entry if it doesn't exist) 1. Create a new job `upsert git tag` in the `tag` stage. Use the code in [this commit](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/90a7045d2eafc3ddf0a462ca2b94795d83096c47/.gitlab-ci.yml) as a starting point. This job is idempotent and will: 1. Add a new CI variable `CREATE_GIT_TAG` (create a git tag if missing) 1. Fetch the most recent version of the analyzer and the description of the change from the `CHANGELOG.md` file. 1. If a git tag matching the version from the `CHANGELOG.md` file does not exist, create a new git tag, provide a tag description which matches the description for the change from the `CHANGELOG.md` file Note: The job is skipped unless `CREATE_GIT_TAG` is true 1. Update the `CI/CD Environment variables` following projects to set `CREATE_GIT_TAG: true` and also configure the `GITLAB_TOKEN` environment variable to match the one in the [ci-templates](https://gitlab.com/gitlab-org/security-products/ci-templates/-/settings/ci_cd) `CI/CD Settings`: - [gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) - [gemnasium-maven](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven) - [gemnasium-python](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python) - [retire.js](https://gitlab.com/gitlab-org/security-products/analyzers/retire.js) - [bundler-audit](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit) ### Future Steps 1. Increase the frequency of the scheduled CI piplines. 1. Create a new MR with the following changes, and include the Static Analysis team to make sure they approve the changes: 1. Rename [`release-major`](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/4ee6b59/includes-dev/docker.yml#L7-7) stage to `release` 1. Rename [`major`](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/4ee6b59/includes-dev/docker.yml#L106-112) to `release major` 1. Rename [`minor`](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/4ee6b59/includes-dev/docker.yml#L113-118) to `release minor` 1. Rename [`tag version`](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/4ee6b59/includes-dev/docker.yml#L87-97) to `release patch` 1. Merge [`release-version`](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/4ee6b59/includes-dev/docker.yml#L6-6) into `release` See [this MR](https://gitlab.com/gitlab-org/security-products/ci-templates/-/merge_requests/270) for a starting point for the above changes. 1. Export the `major`, `minor`, `patch` version numbers as CI variables to be used by the `release` jobs using [dotenv export artifact](https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job) 1. If things go well and Static Analysis agrees with the change, remove the `PUBLISH_IMAGES` and `CREATE_GIT_TAG` variables. By removing these variables, the behaviour of the default branch build will be to automatically create a new git tag if one doesn't exist, and release the images ### Intended users * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) ### Further details This change will eliminate the need for this issue: [Warn when analyzer changelog is updated without a new release](https://gitlab.com/gitlab-org/gitlab/-/issues/338858) ### Documentation - Update the [Versioning and release process](https://gitlab.com/gitlab-org/security-products/analyzers/common#versioning-and-release-process) docs - ~~Update the [Security and Build fixes of Go](https://about.gitlab.com/handbook/engineering/development/secure/release_process.html#git-tag-to-rebuild) docs~~ (not necessary - see [this discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/342986#note_812309162) for more details) ### What does success look like, and how can we measure that? If a developer forgets to release an analyzer, it will automatically be built and released from a scheduled job. We can test this by updating the changelog entry for an analyzer and ensuring that the scheduled job automatically tags and releases a new version. ### What is the type of buyer? ~"Enterprise Applications" ~"GitLab Ultimate" ### Is this a cross-stage feature? This change will affect ~"Category:Dependency Scanning". If ~"Category:SAST" wants to make use of it, they can do so by setting `CREATE_GIT_TAG: "true"` and configuring `GITLAB_TOKEN` to a valid access token. /cc @NicoleSchwartz @gonzoyumo @fcatteau