Support for release key in gitlab-ci.yaml

What does this MR do?

A new node is added to the gitlab-ci.yaml file to support the creation of Releases. This MR will define the nodes added , validate them and make the parsed configuration available to the pipelines. The code to process the configuration will not be included in this MR.

The Release will only be created if the job is processed without error. If an error is returned from Rails API during Release creation, the release job will fail. There can multiple release jobs in a pipeline

release:
  stage: release
  only: tags
  script:
    - make changelog | tee release_changelog.txt
  release:
    name: Release $CI_TAG_NAME
    tag_name: v0.06
    description: ./release_changelog.txt
    assets:
      links:
        - name: cool-app.zip
          url: http://my.awesome.download.site/1.0-$CI_COMMIT_SHORT_SHA.zip
        - name: cool-app.exe
          url: http://my.awesome.download.site/1.0-$CI_COMMIT_SHORT_SHA.exe

These methods are supported:

• tag_name - mandatory
• name - optional
• description - mandatory
• assets - optional
• assets:links:name - optional
• assets:links:url - mandatory if assets specified

release:tag_name

The tag_name must be specified, and this can either refer to a Git tag or can be specified by the user.

If the $CI_COMMIT_TAG predefined variable is specified as a tag name, it is necessary to also specify only: [tags], otherwise the job will fail when running against branches.

When the specified tag doesn't exist in repository, a new tag will be created from the associated ref of the pipeline. In this case, the tag will point to the latest commit of the ref, instead of the associated SHA of the pipeline.

For example, when creating a Release from a Git tag creation:

job:
  release:
    tag_name: $CI_COMMIT_TAG
    description: changelog.txt
  only:
    - tags

It is also possible to create any unique tag, in which case only: tags is not mandatory. A semantic versioning example:

job:
  release:
    tag_name: ${MAJOR}_${MINOR}_${REVISION}
    description: changelog.txt

• The Release will only be created if the main script of the job succeeded.
• If the Release already exists it will not update and will cause the job with release keyword to fail.
• The release section executes after the script tag and before the after_script.

release:name

Specifies the Release name. This is an optional field, if omitted it will be populated with release:tag_name.

release:description

Specifies a file containing the longer description of the Release. This is a mandatory field and can point to a changelog.

release:assets

An array of release:assets:links.

It is possible to add assets to a release, in the form of URLs pointing to an arbitrary external location (such as a S3 bucket or package registry). Any external locations must be populated by a code specified in the script section.

job:
  script:
    - cd src
    - go build hello-world.go
    - make changelog | tee release_changelog.txt
  release:
    name: Release $CI_TAG_NAME
    tag_name: v0.06
    description: release_changelog.txt
    assets:
      - name: hello-world
        url: https://s3.amazonaws.com/myapp_binaries/hello-world

The GitLab Runner will call the GitLab Rails API and create the Release object, adding assets as URLs if defined.

release:assets:links

An array of link entries, optional.

release:
  assets:
    links:
      - name: cool-app.zip
        url: https://downloads.example.org/1.0-$CI_COMMIT_SHORT_SHA.zip

release:assets:links:url

Specifies any valid URL, mandatory if release:assets:links are specified.

release:
  assets:
    links:
      - url: https://downloads.example.org/cool-app.exe

release:assets:links:name

Optional entry. If this is provided, the release:assets:links:url in the same links array element will be displayed using this name, instead of the raw URL.

release:
  assets:
    links:
      - name: cool-app.zip
        url: https://downloads.example.org/1.0-$CI_COMMIT_SHORT_SHA.zip

Tag Only jobs

This functionality has been moved to #118646 (closed)

Feature Flag

This feature will be controlled via a Feature Flag. Refer !19298 (comment 259989805)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• [-] Database guides
• [-] Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• [-] Tested in all supported browsers

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• [-] Label as security and @ mention @gitlab-com/gl-security/appsec
• [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• [-] Security reports checked/validated by a reviewer from the AppSec team

Part of #26013 (closed)

changed milestone to %12.5

added Category:Release Orchestration Deliverable Product Vision FY20 customer devopsrelease [DEPRECATED] direction maturityviable planning priority typefeature workflowin dev + 1 deleted label

changed title from WIP: {-Resolve "Release generation from within .gitlab-ci.yml"-} to WIP: Generation Releases from pipelines

changed the description

assigned to @sean_carroll

added 1 commit

• e2ab4a04 - WIP

Compare with previous version

added 1 commit

• f0ece377 - WIP

Compare with previous version

	1 Warning
	This merge request is quite big (more than 703 lines changed), please consider splitting it into multiple merge requests.

	2 Messages
	This merge request adds or changes files that require a review from the Database team.
	CHANGELOG missing: If this merge request doesn’t need a CHANGELOG entry, feel free to ignore this message. You can create one with: bin/changelog -m 19298 "Support for release key in gitlab-ci.yaml" If you want to create a changelog entry for GitLab EE, run the following instead: bin/changelog --ee -m 19298 "Support for release key in gitlab-ci.yaml" Note: Merge requests with ~backstage, ci-build, meta do not trigger this check.

This merge request requires a database review. To make sure these changes are reviewed, take the following steps:

1. Ensure the merge request has database and databasereview pending labels. If the merge request modifies database files, Danger will do this for you.
2. Prepare your MR for database review according to the docs.
3. Assign and mention the database reviewer suggested by Reviewer Roulette.

Reviewer roulette

Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.

To spread load more evenly across eligible reviewers, Danger has randomly picked a candidate for each review slot. Feel free to override this selection if you think someone else would be better-suited, or the chosen person is unavailable.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not (yet?) automatically notify them for you.

Category	Reviewer	Maintainer
backend	Arthur Evstifeev (@aevstifeev)	Imre Farkas (@ifarkas)
database	Tiger Watson (@tigerwnz)	Mayra Cabrera (@mayra-cabrera)

Generated by Danger

added 1 commit

• b3e1fda2 - WIP

Compare with previous version

added database databasereview pending labels

changed title from WIP: Generation Releases from pipelines to WIP: Support for release key in gitlab-ci.yaml

changed the description

mentioned in issue #26013 (closed)

added 1 commit

• 3ed5a072 - Add release node to gitlab-ci.yaml

Compare with previous version

added 1 commit

• d421156c - WIP

Compare with previous version

added 1 commit

• d0c070d1 - Add release node to gitlab-ci.yaml

Compare with previous version

@dosuken123 this is still a WIP, but could you review to see if I am on the right path here.

I'm not really clear why bundle exec rspec spec/lib/gitlab/ci/yaml_processor_spec.rb:1058 is failing. If you have any insight that would be appreciated.

assigned to @dosuken123

assigned to @ayufan

unassigned @dosuken123

unassigned @ayufan

It seems already reviewed by kamil so I will not go through for now. Feel free to assign me back when you need review from my side.

changed the description

added 1 commit

• ec85a422 - WIP

Compare with previous version

added 1 commit

• 39058b50 - Add release key to gitlab-ci.yaml

Compare with previous version

@ayufan I've updated this MR based on the modified release: structure as discussed. It needs some more tests (I will add them). But, I'm blocked on one thing and was wondering if you could look over it?

• I used descr instead of description as it was conflicting with Gitlab::Config::Entry::Node. I considered alias_method but it wasn't working (perhaps as it is mixed in?), but in any case I think this will cause other problems as downstream code will be expecting description. Do you have any suggestions?

alias_method :description, :title
remove_method :description 

Many thanks!

assigned to @ayufan

added missed:12.5 label

added 1 commit

• b1c59521 - Add release key to gitlab-ci.yaml

Compare with previous version

changed milestone to %12.6

mentioned in commit c945b68e

added 1 commit

• c945b68e - Add release key to gitlab-ci.yaml

Compare with previous version

unassigned @ayufan

added 1 commit

• 64b40a7b - Add changelog

Compare with previous version

marked the checklist item Changelog entry as completed

marked the checklist item Documentation created/updated or follow-up review issue created as completed

marked the checklist item Code review guidelines as completed

marked the checklist item Merge request performance guidelines as completed

marked the checklist item Style guides as completed

mentioned in commit c08d09c6

added 1 commit

• c08d09c6 - Add release key to gitlab-ci.yaml

Compare with previous version

mentioned in commit 84bf790b

added 1 commit

• 84bf790b - Add release key to gitlab-ci.yaml

Compare with previous version

added 1 commit

• 019a7b92 - Additional tests

Compare with previous version

mentioned in commit 309588d4

added 2195 commits

• 019a7b92...c3f023eb - 2193 commits from branch master
• 309588d4 - Add release key to gitlab-ci.yaml
• 46921809 - Additional tests

Compare with previous version

added 1 commit

• 1a033efb - Correct typo

Compare with previous version

added 1 commit

• 4d075d5a - WIP

Compare with previous version

mentioned in commit 7b06e474

changed the description

added 235 commits

• 4d075d5a...14eb7a47 - 234 commits from branch master
• 7b06e474 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit 048047f4

added 1 commit

• 048047f4 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit 78369787

added 92 commits

• 048047f4...f8e60ae6 - 91 commits from branch master
• 78369787 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in merge request !20632 (merged)

mentioned in commit 39a76f2a

added 1 commit

• 39a76f2a - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit b320eec1

added 1 commit

• b320eec1 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit 18e2e689

added 1 commit

• 18e2e689 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit dcd771fc

added 1 commit

• dcd771fc - Support for release key in gitlab-ci.yaml

Compare with previous version

changed the description

resolved all threads

unmarked as a Work In Progress

@dosuken123 assigned to you for review.

Please note the description conflict still exists. I have temporarily changed the entry name to desc, as removing Attributable meant that the lib/gitlab/ci/config/entry/release.rb spec was failing, as well as others, due to redefinition.

Not sure what the best way forward on this is, suggestions welcome.

assigned to @dosuken123

mentioned in commit bbdce289

added 184 commits

• dcd771fc...fede9077 - 183 commits from branch master
• bbdce289 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit 7122ec7a

added 181 commits

• bbdce289...26886d56 - 180 commits from branch master
• 7122ec7a - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit cc9eb4c3

added 21 commits

• 7122ec7a...38bd0126 - 20 commits from branch master
• cc9eb4c3 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit 3d73f7fb

added 1 commit

• 3d73f7fb - Support for release key in gitlab-ci.yaml

Compare with previous version

Hi @dosuken123 I think this is good to go now, could you please review.

@sean_carroll Thanks. Overall LGTM. I left a few comments, but probably an important question is !19298 (comment 251749031). I think we should have tag or tag_name keyword under release, as it's consistent with Release API format. Let me know what you think.

unassigned @dosuken123

added 1 commit

• 96bbaaa9 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb

Compare with previous version

added 1 commit

• e6986a3e - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb

Compare with previous version

added 2 commits

• d3490842 - Remove changelog as suggested
• a45180c6 - Apply reviewer suggestions

Compare with previous version

added workflowin review label and removed workflowin dev label

changed the description

added 1 commit

• d024491a - Rename tag to tag_name

Compare with previous version

mentioned in commit 95210fa0

added 1352 commits

• d024491a...a36d844f - 1346 commits from branch master
• 95210fa0 - Support for release key in gitlab-ci.yaml
• 65bf9e16 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• 7c82745e - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• 7caad42c - Remove changelog as suggested
• a24bcc95 - Apply reviewer suggestions
• cc528118 - Rename tag to tag_name

Compare with previous version

mentioned in commit 0860d055

added 45 commits

• cc528118...1326e4fd - 39 commits from branch master
• 0860d055 - Support for release key in gitlab-ci.yaml
• b13cdd55 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• fc198e59 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• d200911d - Remove changelog as suggested
• 3ac9d7dc - Apply reviewer suggestions
• 4f6f392e - Rename tag to tag_name

Compare with previous version

added 1 commit

• 1d89244e - Fix release spec

Compare with previous version

added 1 commit

• 65ea90ca - Add rules for CI_COMMIT_TAG

Compare with previous version

added 1 commit

• a661763e - Fix linter issues

Compare with previous version

mentioned in commit 53f94602

added 203 commits

• a661763e...5f56d89c - 194 commits from branch master
• 53f94602 - Support for release key in gitlab-ci.yaml
• 9fffc955 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• 9c7cfaef - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• 3118c694 - Remove changelog as suggested
• afdf5ee8 - Apply reviewer suggestions
• 2967c6e3 - Rename tag to tag_name
• 7f1862ce - Fix release spec
• 50fd0237 - Add rules for CI_COMMIT_TAG
• b4c3835c - Fix linter issues

Compare with previous version

added 1 commit

• 55da6502 - Change release to symbol

Compare with previous version

added 1 commit

• 9224b568 - Added only tags

Compare with previous version

mentioned in commit 68379d50

added 45 commits

• 9224b568...71da2ba2 - 34 commits from branch master
• 68379d50 - Support for release key in gitlab-ci.yaml
• 7d3bff25 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• 13ccf2d6 - Apply suggestion to spec/services/ci/create_pipeline_service_spec.rb
• 7908d8b6 - Remove changelog as suggested
• f13c7937 - Apply reviewer suggestions
• 32bea7e3 - Rename tag to tag_name
• fb5b840a - Fix release spec
• 1a8015d9 - Add rules for CI_COMMIT_TAG
• 89734ff7 - Fix linter issues
• 8252a87c - Change release to symbol
• 845e0413 - Added only tags

Compare with previous version

added 1 commit

• 753a3d87 - Added only: :tags check

Compare with previous version

mentioned in commit bc191bbf

added 1 commit

• bc191bbf - Support for release key in gitlab-ci.yaml

Compare with previous version

added 1 commit

• 90f79f57 - Fix linter error

Compare with previous version

added 1 commit

• 4039970b - Fix linter errors

Compare with previous version

Added support for $CI_COMMIT_TAG

Assigned to @dosuken123 for review.

assigned to @dosuken123

@sean_carroll Thanks. It looks nice One concern is the evaluation for $CI_COMMIT_TAG. It's actually more tricky than the current code, so that I'd say we should improve in the future MR than tackling the problem in this MR.

Also, we might want to use feature flag to disable the feature by default until the whole feature is complete (Please take a look at https://about.gitlab.com/handbook/engineering/development/ci-cd/release/#issue-template-for-a-feature-development). This lets us avoid unexpected behavior from the incomplete feature on production server. In this feature's case, I'd suggest something like this

module Seed
  class Build < Seed::Base
    def initialize(pipeline, attributes, previous_stages)
      @resource_group_key = attributes.delete(:resource_group_key)
      attributes.fetch(:options, {})&.delete(:release) unless Feature.enabled?(:ci_release_generation, @project)

(We might be able to check the flag in Config::Entry level, instead)

unassigned @dosuken123

One concern is the evaluation for $CI_COMMIT_TAG. It's actually more tricky than the current code, so that I'd say we should improve in the future MR than tackling the problem in this MR.

@dosuken123 I'm not clear on this. Do you mean to split support for $CI_COMMIT_TAG out?

added 1 commit

• e9772324 - Support embedded $CI_COMMIT_TAG

Compare with previous version

changed the description

changed the description

mentioned in issue #118439

Summary of scope clarifications:

• Move config to it's own table, #118439
• Control this feature with a Feature flag
• Expand only: ['tags'], possibly in a new MR.

added 1 commit

• 771430f1 - Apply reviewer suggestion

Compare with previous version

added 1 commit

• bcd53aa7 - Apply reviewer suggestion

Compare with previous version

added 1 commit

• af533464 - Apply reviewer suggestions

Compare with previous version

added 1 commit

• d15ab59c - Fix linter issues

Compare with previous version

added 1 commit

• 344a819c - Apply reviewer suggestion

Compare with previous version

@dosuken123, based on today's call and the comments about $CI_COMMIT_TAG !19298 (comment 260879118), then to complete this MR two things are needed:

• Remove the logic surrounding validation of $CI_COMMIT_TAG and create a new issue to track this, probably via build.tag? or pipeline.tag?
• Wrap this code in a Feature flag !19298 (comment 259989805)

WDYT ?

resolved all threads

added 1 commit

• b4c45fc5 - Add feature flag

Compare with previous version

mentioned in commit 36dd9092

added 547 commits

• b4c45fc5...467b0e70 - 538 commits from branch master
• 36dd9092 - Support for release key in gitlab-ci.yaml
• a2e681de - Fix linter errors
• 1bf6b768 - Support embedded $CI_COMMIT_TAG
• f182b6eb - Apply reviewer suggestion
• 00f76ec2 - Apply reviewer suggestion
• 2204863b - Apply reviewer suggestions
• a2f75a11 - Fix linter issues
• 026bec14 - Apply reviewer suggestion
• 72edb77d - Add feature flag

Compare with previous version

mentioned in issue #118646 (closed)

mentioned in commit 8e49e22a

added 1 commit

• 8e49e22a - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit f9be13f7

mentioned in commit 68d8bd78

added 1 commit

• 68d8bd78 - Support for release key in gitlab-ci.yaml

Compare with previous version

changed the description

Tag-only jobs validation has been moved to #118646 (closed)

added 101 commits

• 467b0e70...5a6ebe09 - 101 commits from branch master

Compare with previous version

Assigned to @dosuken123 for review.

assigned to @dosuken123

mentioned in commit 270dea95

mentioned in commit 9a77c3e5

added 201 commits

• 5a6ebe09...b5832553 - 200 commits from branch master
• 9a77c3e5 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit 92fc6934

added 1 commit

• 92fc6934 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit d2bfa168

added 1 commit

• d2bfa168 - Support for release key in gitlab-ci.yaml

Compare with previous version

added 1 commit

• 2f8e4e31 - Remove merge_trains_enabled column from project_ci_cd_settings table

Compare with previous version

mentioned in commit 2a95a1e7

added 1 commit

• 2a95a1e7 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit d97d6d37

added 98 commits

• 2a95a1e7...cd688518 - 97 commits from branch master
• d97d6d37 - Support for release key in gitlab-ci.yaml

Compare with previous version

mentioned in commit ffdf2421

added 1 commit

• ffdf2421 - Support for release key in gitlab-ci.yaml

Compare with previous version

resolved all threads

approved this merge request

@sean_carroll Thanks. It looks good to me I left one nitpick, but it's not big deal.

unassigned @dosuken123

added 1 commit

• 27cbf6dd - Apply reviewer suggestion

Compare with previous version

resolved all threads

mentioned in commit 039e23e8

added 1 commit

• 039e23e8 - Support for release key in gitlab-ci.yaml

Compare with previous version

Assigned to @ayufan for maintainer approval.

assigned to @ayufan

resolved all threads

changed milestone to %12.7

added missed:12.6 label

added missed-deliverable label

mentioned in commit 14c4f9ea

added 565 commits

• 039e23e8...403b60b1 - 564 commits from branch master
• 14c4f9ea - Support for release key in gitlab-ci.yaml

Compare with previous version

@ayufan I just rebased this against master. It is ready for maintainer review. Thanks!

added 1 commit

• 0b03005e - Fix linter issue

Compare with previous version

mentioned in commit 234ebded

added 443 commits

• 0b03005e...9e71803b - 441 commits from branch master
• 234ebded - Support for release key in gitlab-ci.yaml
• 24bebecc - Fix linter issue

Compare with previous version

Rebased against master.

@sean_carroll

Thanks this looks good. I only have one suggestion.

I think that this is a first step, as it misses the persisting of the release: anywhere. We only parse it.

Can we somehow comment it in either code, or as part of separate issue?

added 1 commit

• dda14c8b - Apply maintainer suggestion

Compare with previous version

mentioned in commit b249b751

added 1 commit

• b249b751 - Support for release key in gitlab-ci.yaml

Compare with previous version

Thanks @ayufan

I think that this is a first step, as it misses the persisting of the release: anywhere. We only parse it.

I'm not clear on that, I would have expected it to be already persisted in ci_builds.options with the rest of the .gitlab-ci.yaml.

I have opened a new issue to handle the persistence. Could you possibly point me to where persistence is handled for the rest of the yaml?

mentioned in issue #196110 (closed)

resolved all threads

Assigned to @ayufan for maintainer review.

resolved all threads

mentioned in issue #196171 (closed)

approved this merge request

merged

mentioned in commit 62914e26

mentioned in commit 0130c0fb

removed direction label

mentioned in merge request gitlab-com/www-gitlab-com!39076 (merged)

mentioned in issue #207842 (closed)

mentioned in merge request !26385 (merged)

mentioned in merge request !32528 (closed)

mentioned in merge request !48365 (merged)

mentioned in issue release-cli#128