Skip to content

Release generation from `.gitlab-ci.yml` via API call

Problem to solve

In issue, #26013 (closed) , we create the configuration for releases in the .yml and exposed the endpoints. In order for a release to be created, we will need to support the generation of a Release object in Rails, via an API call.

The purpose of this issue is to :

  • Identify where the API call to create the Release will be issued from (Runner, Rails or other?)
  • Develop the code to call the API.

Please note:

  • This issue supports the creation of assets as urls, but not binary assets. Binary assets are still supported in #36133 (closed)
  • Documentation MR !19237 (merged) will be merged when this issue is completed.

Target audience

Automation engineers who are working on releases and want a native way to add releases without using script: elements to curl an API

Further details

This is good polish for the releases feature in that it makes the integration feel better integrated into the product. Also, this provides better overall experience once features like https://gitlab.com/gitlab-org/gitlab-ce/issues/56023 are implemented.

Goals

In this iteration the focus will be on:

  • automated creation of the Release object as a CI step using a new releases-cli, using the existing Releases API
  • keeping the solution decoupled and extensionable for future work
  • allowing the releases-cli to be used by any process which has access to Job-Token, for example third-party CI.
  • Support the addition of Upcoming Releases in future.

Assumptions

  • we want to have a MVC in 12.9
  • the .gitlab-ci.yml structure is defined in #26013 (closed)
  • the majority of the development will be done by the Release team, with support from the Runner team if needed
  • The CLI will only use Job-Token auth, although this could be extended in future if desired.

Proposal

MVC

The MVC supports creation of a Release object in Rails, based on definitions in .gitlab-ci.yml. The Release is created with name and description. Description can be passed from a file inside the repo, eg: CHANGELOG.md

release_upload:
  script:
    - gitlab-runner release-upload --name="My Release" --description-file=CHANGELOG.md

Architecture

A new component, the Release CLI will be developed in Go. This is a decoupled utility than can be called by the Runner, by a third-party CI or directly from the command-line. It uses Job-Token auth only in this iteration.

The Rails application will transform the .gitlab-ci.yml into CLI commands. The CLI will construct the API call and call the Rails API to create the release.

The CLI can also be called independently, and can still create the Release via Rails API if the Job-Token and correct command line params are provided.

sequenceDiagram
  participant Rails
  participant ReleaseCLI
  participant Runner
  Runner->>Rails: 1. Runner calls API for job info
  Rails->>Rails : 2. Yaml exposed as Steps
  Runner->>ReleaseCLI : 3. Runner calls CLI on Job success
  ReleaseCLI->>Rails : 4. CLI retrieves Release Steps
  ReleaseCLI->>Rails : 5. CLI creates Release
1. Runner calls API for job info

The Runner polls Rails for new Jobs. This is the current process and changes are needed here.

2. Yaml exposed as Steps

The release node of the .gitlab-ci.yaml configuration is converted into Steps and made available via API endpoint.

3. Runner calls CLI on Job success

The Runner executes the Job, and upon success calls the Release CLI. The evaluation of if the Release must be created is made by the CLI, so this step implies the Runner always calls the Release CLI at the end of a successful Job.

  • Is this correct ? @steveazz
4. CLI retrieves Release Steps

The Release CLI calls the Rails API to retrieve the release configuration (as Steps).

5. CLI creates Release

The Release CLI makes an API call to Rails to create the Release.

Other Considerations

The work is across three applications, and this could be split into 3 issues and multiple MRs.

Release CLI (Issue 1)

  • To be coded in Go and belongs to the Release team
  • This is a new GitLab component, will be called the Release CLI and will be defined in a new project #38352 (comment 262264631)
  • What are the command line params that will be provided ? #38352 (comment 262266271)
  • We can use the artifacts CLI code as an example starting point, and this PoC also exists

Rails (Issue 2)

  • Rails already has been extended to support the release node in .gitlab-ci.yml and the API has been extended to permit Job-Token access to the Release POST endpoint
  • The .gitlab-ci.yml needs to be transformed into Runner CLI steps inside of Rails #38352 (comment 262268206)

Runner (Issue 3)

  • Shared code is extracted from the Runner into a module
  • The Runner calls the Release CLI at the end of a successful Job

Third-Party access

  • Test coverage on the Runner CLI will need to support third-party CI and command line execution

Permissions and Security

  • All the work supported in the Job Token will translate to this issue

Documentation

What does success look like, and how can we measure that?

  • Usage of the release functionality from yml

What is the type of buyer?

  • Community Edition

Links / references

Edited by Sean Carroll