Documentation for "Safe Deployments"
Problem
We have a bunch of safe mechanizms/features for safe deployments, such as:
- Limit the pipeline job concurrency
resource_group
- Disallows deployments for old versions
- Deploy Freeze
However, we don't have central documentation at this moment.
This confuses users sometimes, for example,
I'm experiencing the same issue here. It's unclear to me what part of my configuration is causing this error. I would like to address this properly without enabling the
Skip older, pending deployment jobs
feature. More information on the error would be greatly appreciated.
I just ran into and was confused by this too, can we make it more clear in the error message what is causing this?
The problem is in the same environment in both jobs. It's very strange for me, because it's normal in other ci/cd's. There are a lot of cases when it's needed. And, yes, i know about wildcard environments.
Reference: #209866 (closed)
We should improve the documentation and UI.
Proposal
So for example, we can document like:
Sequential Deployment (or Safe Continuous Deployment)
If your project follows Continuous Deployment practice that deploys master
branch to production
environment with GitLab CI/CD pipelines, you might encounter the following problems due to the asynchronous nature of pipeline jobs*:
- Multiple deployment jobs run concurrently to the same environment. It makes the environment unstable because deployment script conflicted and finished in immature state.
- An old deployment job overwrites the latest deployment. It results in an unintentional rollback. Users experience old feature sets on the
production
website even though you see that the latest deployment job successfully finished.
To address these problems, GitLab provides the following options:
- Limit the pipeline job concurrency.
- Disallows deployments for old versions.
Limit the pipeline job concurrency
You can limit the pipeline job concurrency by specifying resource_group
keyword into your .gitlab-ci.yml. With this option, your deployment jobs run once at a time therefore you can effectively avoid deployment job conflicts to the same environment. For example,
- Pipeline-A starts running with SHA-A.
- Pipeline-B starts running with SHA-B (newer).
- Pipeline-A starts a deployment.
- Pipeline-B waits for the Pipeline-A's deployment is done.
(and more explanation ...)
Documentation: https://docs.gitlab.com/ee/ci/yaml/#resource_group
Disallows deployments for old versions.
You can disallows deployments for old versions by enabling Skip outdated deployment jobs checkbox. With this option, when an old deployment job is about to run, the job fails as "The deployment job is older than the previously succeeded deployment job, and therefore cannot be run". For example,
- Pipeline-A starts running with SHA-A.
- Pipeline-B starts running with SHA-B (newer).
- Pipeline-B finishes. Now SHA-B is on the production environment.
- Pipeline-A fails as it's going to deploy SHA-A to the production.
(and more explanation ...)
Documentation: https://docs.gitlab.com/ee/ci/pipelines/settings.html#skip-outdated-deployment-jobs
Troubleshooting
Having multiple deployment jobs in the same pipeline results in "The deployment job is older than the previously succeeded deployment job..." error
...