Skip to content

CI architecture changes

I thought about the possible improvements and how it fits our ideas about how we should handle testing and deployments, including the workflows and data models.

Overview

We want one unified interface for CI and CD, it will all happen in .gitlab-ci.yml and there is no separation between testing and deployment. Deployments can have many stages and jobs

  • CiCommit is a commit in the repo
  • A CiCommit contains many jobs
  • Jobs might add metadata for code coverage
  • ExternalJob is for external CI services that set the commit status
  • Build is done one or more times for a job with a retry, unit for Runners
  • Request is a way to kick off another project (call it trigger?)

Request is a group of jobs running in the same environment (same commit, same reference), a push will trigger a new request, without this we need to store the trigger, type and ref in duplicate in the job. One CiCommit can have multiple request, for example new tag, new branch, new MR.

Data models

CiCommit:
- many Jobs
- many Requests
- Status: summary of all Requests

Job: (currently CommitStatus)
- Name/Stage
- Created/Started/Finished
- Status
- Author
- Request

ExternalJob < Job (currently GenericCommitStatus)
- TargetURL
- Description
- DownloadURL (future)
- RetryURL (future)
- CancelURL (future)

Build < Job (currently Ci::Build)
- Trace
- Artifacts

Request (currently TriggerRequest)
- Type: Push, Tag, Trigger, MergeRequest, Manual
- Ref/Sha/BeforeSha
- Status (pending, running, succeeded, failed, canceled)
- Trigger (optional)
- Author
- many Jobs

UX

We will show a list of all requests with information for what event it was created: push, tag, trigger, merge request, manual.

Workflows

1. Git tag/push:

There's no UI for this, only low-level implementation.

  • If .gitlab-ci.yml is present: create Request object from webhook
  • If no .gitlab-ci.yml: create Request object in skipped state
  • Execute Request Hooks
  • Gather all external statuses
  • Execute Request Hooks when state changes

2. Trigger externally for specific environment

There's no UI for this, only low-level implementation.

  • Create request object from trigger payload
  • Execute Request Hooks
  • Create builds from .gitlab-ci.yml (if present), otherwise start in pending state

3. Manual triggers from UI

  • We present the button to trigger builds for specific commit or branch
  • We present an UI where we ask for: -- variables that will be injected in build (we pre fill it with previously passed variables) -- we ask about environment, default selected by default -- we ask to write description
  • User clicks Start/Create or Deploy (we don't know if this is deployment or not, so I would rather use something that doesn't say that this is always deployment): in some cases it will be just re-testing of the change, in some it will be build, test and deploy, in some it will be just deploy (depends on .gitlab-ci.yml and on external services that will consume Request Hooks)

The Status API:

When status is posted we create ExternalJob object in context of all Requests that are matched by ref parameter.

We introduce a new parameter: request_id to uniquely identify the request. Ideally the other CI services should use the request_id to post the status only in context of specific request.

Status parameters:

  1. sha - commit SHA,
  2. ref - branch or tag (optional),
  3. request - request ID from Trigger API (used to filter the triggers),
  4. state - success, failure, running, pending, canceled,
  5. description - information about the status,
  6. target_url - URL where detailed information about the status can be found,

The Trigger API

The Trigger API will be also used to trigger deployments, builds or just tests.

We will allow to specify environment. The external services should check environment and ignore the hook if they don't want to deployment or check for specific environment.

Trigger parameters:

  1. ref - branch, tag or sha,
  2. description - short description of the trigger,
  3. variables - passed with Request Hooks and passed to all builds,

We create an new Request and fill it with all received parameters.

Reason for having or not having the environments

I thought whether we need environments or not. When we start implementing it we can skip the separate environments and just build everything around the branches. We will have gitlab-ci.yml where we will create jobs that will get executed for different branches, so our environments will be tied to the branches or tags.

You push to staging and this branch gets deployed to the staging environment.

However, as later improvement I see a benefit to introduce a environments, because it will allow us to have separate credentials (the production can use different credentials). This will make it possible to better track different environments and versions deployed to them. From my perspective the concept of environments should be span on multiple projects allowing you to have a group of resources (runners, credentials) that share the data and that we can visualise it better.

At this moment introducing separate environments will only complicate the whole solution. Our users already are used to use different branches to trigger deployments to different environments. This concept is pretty well described and easy to understand.