Skip to content
Snippets Groups Projects

Create Deployments for processed (ready-to-run) jobs only

    • Closed Issue created by Shinya Maeda @shinya.maeda

    Problem

    Currently, we create Deployment for all deployment jobs. Since Deployment is a stateful model, this looks fine at the glance, however, it doesn't make sense to create a Deployment for jobs that are not executable state yet. Because of this, there are some data integrity and UX issues caused, such as:

    • Users seeing Deployment Approval even if it's not executable yet.
    • Users seeing manual deployment jobs in Environment details page even if it's not executable yet.
    • Irrelevant Deployment records in Environment page harms the readability on the page.

    Technical side

    Basically, there are two steps until a job becomes executable state:

    1. Create a job. https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/create_pipeline_service.rb. Job status is created.
    2. Process a job. https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/pipeline_processing/atomic_processing_service.rb. Job status transitions to pending (pickable by runner), manual (manually executable by user) OR stay created if the pipeline pointer hasn't reached the job yet.

    Example

    You have the following .gitlab-ci.yml:

    stages:
  - stage-1
  - stage-2
  - stage-3

deploy-a:
  stage: stage-1
  script: echo
  environment: prd
  when: manual
  allow_failure: false

deploy-b:
  stage: stage-2
  script: echo
  environment: prd
  when: manual
  allow_failure: false

deploy-c:
  stage: stage-3
  script: echo
  environment: prd
  when: manual

    Expected behavior

    1. deploy-a (manual with deployment) => deploy-b (created w/o deployment) => deploy-c (created w/o deployment)
    2. deploy-a (success with deployment) => deploy-b (manual with deployment) => deploy-c (created w/o deployment)
    3. deploy-a (success with deployment) => deploy-b (success with deployment) => deploy-c (manual with deployment)

    Current bug behavior

    1. deploy-a (manual with deployment) => deploy-b (created with deployment) => deploy-c (created with deployment)
    2. deploy-a (success with deployment) => deploy-b (manual with deployment) => deploy-c (created with deployment)
    3. deploy-a (success with deployment) => deploy-b (success with deployment) => deploy-c (manual with deployment)

    Technical proposal

    1. Execute ::Deployments::CreateForBuildService.new.execute(processable) right before Ci::ProcessBuildService runs in PipelineProcessing::AtomicProcessingService. Ideally, we should run inside but this causes a PreventCrossDatabaseModification error because ci_* tables (CI DB) and deployments (Main DB) runs in the same transaction due to Gitlab::OptimisticLocking.
    2. Remove create_deployments! from InitialPipelineProcessWorker. This is temporary fix until this proper fix is placed.
    3. Remove ::Deployments::CreateForBuildService from RetryJobService as it's not necessary. The one in the PipelineProcessing::AtomicProcessingService can cover all paths.

    See PoC for more information.

    NOTE: We need to fix Auto Job Retry is performed in a DB transaction... (#390638 - closed) at first, otherwise we'll encounter PreventCrossDatabaseModification error (because Deployments::CreateForBuildService will run in the same transaction with ci_builds)

    At any rates, we should use a feature flag to rollout the change gradually.

    Edited by Shinya Maeda

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply