Skip to content

GitLab Next

Project 'gitlab-org/gitlab-ee' was moved to 'gitlab-org/gitlab'. Please update any links and bookmarks that may still have the old path.
Closed
Created by Fabio Busatto@bikebillyContributor

Incremental rollouts (manual)

Description

When you have a new version of your app to deploy, you may want to use incremental rollouts to replace just a few pods with the latest code. This will allow you to check how it is going, and then increase up to 100%

We should provide such flow in Auto DevOps, so users can leverage incremental rollouts easily.

Owners

BE: @mayra-cabrera

Proposal

Allow users to specify if they want to use incremental rollouts for their production deployments via the $INCREMENTAL_ROLLOUTS_ENABLED variable.

If yes, instead of the standard production job different jobs will be created.

  1. rollout 10%
  2. rollout 25%
  3. rollout 50%
  4. rollout 100%

Job definitions are something similar to this:

.production: &production
  stage: production
  script:
    - echo "Standard Deploy"
    - deploy
  environment:
    name: production
    url: http://$CI_PROJECT_PATH_SLUG.example.com
  only:
    refs:
      - master

production:
  <<: *production
  except:
    variables:
      - $INCREMENTAL_ROLLOUT_ENABLED

rollout 10%:
  <<: *production
  script:
    - echo "Incremental Deploy 10%"
    - deploy rollout 10
  only:
    variables:
      - $INCREMENTAL_ROLLOUT_ENABLED

rollout 25%:
  <<: *production
  script:
    - echo "Incremental Deploy 25%"
    - deploy rollout 25
  when: manual
  only:
    variables:
      - $INCREMENTAL_ROLLOUT_ENABLED

rollout 50%:
  <<: *production
  script:
    - echo "Incremental Deploy 50%"
    - deploy rollout 50
  when: manual
  only:
    variables:
      - $INCREMENTAL_ROLLOUT_ENABLED
    
rollout 100%:
  <<: *production
  script:
    - echo "Incremental Deploy 100%"
    - deploy rollout 100
  when: manual
  only:
    variables:
      - $INCREMENTAL_ROLLOUT_ENABLED

The first one (10%) will be auto, the other three will be when: manual. The last three will automatically appear in the dropdown for the production environment

When an incremental rollout job is done, it should:

  1. scale the new version to the required percentage
  2. scale the old version to reach the total number of pods required by $REPLICAS

Possible edge cases

  1. when 100% is reached, scaling down the rollout should be avoided
  2. since we can have staging, both production and rollout 10% should be duplicated to allow auto/manual flow
  3. redeploy button should consider incremental rollouts when choosing the job to run
Edited by Fabio Busatto