Skip to content
Snippets Groups Projects
Commit fb5d5dce authored by Robert Speicher's avatar Robert Speicher
Browse files

Merge branch 'rs-doc-updates' into 'master'

Documentation cleanup

See merge request !421
parents 889bff97 86cc5e12
No related branches found
No related tags found
1 merge request!421Documentation cleanup
......@@ -2,10 +2,12 @@
Building artifacts for public consumption is handled on our dev instances via
[`omnibus-gitlab`](https://dev.gitlab.org/gitlab/omnibus-gitlab). All builds
are initiated via some sort of trigger:
* Nightly pipelines are kicked off via a Cron Job
* Nightly pipelines are kicked off [via a Cron Job][nightlies]
* Builds of tagged packages are automatically started when that tag is pushed
into the repo
[nightlies]: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules
# Nightly Builds
## Pipeline
......
# Overview
The [auto-deploy process overview](auto-deploy.md) provides details on the
building pieces of the process, a schedule overview and answers a few frequently
asked questions.
Previous contents of this document were used to transition from the monthly deploy cadence to the new deploy system. For the most current information,
checkout the [Releases Handbook page](https://about.gitlab.com/handbook/engineering/releases/).
## Overview
# Overview
GitLab.com deployments are initiated by GitLab Chatops, this triggers
GitLab.com deployments are initiated by GitLab Chatops, which triggers
a new pipeline on the [Deployer](https://ops.gitlab.net/gitlab-com/gl-infra/deployer) project.
To see all available options for deploy run `/chatops run deploy --help`:
To see all available options for deploy run `/chatops run deploy --help`
## Creating a new deployment for _upgrading_ GitLab
Deployments are initiated using GitLab Chatops.
For example, to initiate a deployment of 11.8.0 to **staging**:
For example, to initiate a deployment of 11.8.0 to **staging** (default):
```
/chatops run deploy 11.8.0.ee.0
......@@ -17,7 +17,7 @@ For example, to initiate a deployment of 11.8.0 to **staging**:
## Override variables
Variables can be set on the deployer pipeline to change its behavior or
the jobs that will be executed. See [variables.md](../../runbooks/variables.md)
the jobs that will be executed. See [variables.md](../../runbooks/variables.md).
### Bypassing Failures
......@@ -63,28 +63,13 @@ the forced failure on the prepare job by using the option
This essentially sends the variable `PRECHECK_IGNORE_ERRORS` to the deployment
pipeline.
### Creating a new deployment for _rolling back_ GitLab
### Creating a new deployment for _rolling back_ GitLab to an earlier version
**IMPORTANT: Before initiating a rollback ensure to follow the [appropriate runbook](../../runbooks/rollback-a-deployment.md),
the following paragraph covers how to start a manual rollback pipeline on the deployer project in case of a chatops failure.**
Rollbacks are covered in detail in the [rollback runbook](../../runbooks/rollback-a-deployment.md).
In case of need it's possible to start a deployer pipeline by initiating a new pipeline
on the [deployer project](https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/new)
**default branch** and setting the environment and version CI variables.
## CI/CD Pipeline for a Deployer
* `DEPLOY_ENVIRONMENT`: **required** (i.e.: `gstg`, `gprd`)
* `DEPLOY_VERSION`: **required**
* `DEPLOY_ROLLBACK`: **required for rollback**, set to `true` to initiate a rollback pipeline
_Note that all pipelines initiated on branches will automatically set the
`--dry-run` flag which can be useful for testing._
![Deployer Pipeline](../images/deployer-example.png)
### CICD Pipeline for a Deployer
#### Assets
### Assets
Assets are either extracted from the assets docker image if it is available or pulled from the omnibus package.
This is done with a [shell script](https://gitlab.com/gitlab-com/gl-infra/deploy-tooling/-/blob/master/bin/fetch-and-upload-assets) in the deploy pipeline at the same time when we run database migrations in a job called `<ENV>-assets. See [.gitlab-ci](https://gitlab.com/gitlab-com/gl-infra/deployer/-/blob/d0b25b4c783d6010a9d7d0ce64ae49cdda813bb6/.gitlab-ci.yml#L345-359).
......@@ -116,7 +101,7 @@ When the browser requests an asset under `/assets`, it will either be for `asset
There is an outstanding issue to simplify this by removing the Fastly CDN https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11411
#### Upgrade Pipeline (default)
#### Upgrade Pipeline
The following stages are run for upgrade for a single environment upgrade:
......@@ -143,30 +128,6 @@ graph LR;
f ==> g>gitlab-qa];
```
#### Upgrade Pipeline (multi-environment)
Multiple environments can be passed into the deployer project to chain
deployments, this works by setting `DEPLOY_ENVIRONMENT` to a comma separated
list of environments. The following chained environments are supported:
* `DEPLOY_ENVIRONMENT=gstg-cny,gstg,gprd-cny`: Staging canary -> Staging -> Production canary
* `DEPLOY_ENVIRONMENT=gstg-cny,gstg,gprd-cny,gprd`: Staging canary -> Staging -> Production canary -> Production
* `DEPLOY_ENVIRONMENT=gstg,gprd-cny`: Staging -> Production canary
* `DEPLOY_ENVIRONMENT=gstg,gprd-cny,gprd`: Staging -> Production canary -> Production
A deployment pipeline from staging -> production as the following stages that
provide safety QA checks and a manual promotion to production:
```mermaid
graph LR;
a>Staging deploy] ==> b>Staging QA smoke tests];
b ==> c>Production canary deploy];
c ==> d>Canary smoke tests];
d ==> e>Manaul approval];
e ==> f>Production Deploy];
```
#### Change-lock periods for deployment
Before deploying to stage that receives production traffic there is an additional
......
......@@ -27,22 +27,6 @@ Traffic is divided by the following backends:
* pages: anon requests to a pages site
* https_git: anon https request to git-upload-pack across various repositories.
### GitLab-QA
GitLab-QA is configured to run every 10 minutes in a [scheduled
job](https://ops.gitlab.net/gitlab-com/gl-infra/traffic-generator/pipeline_schedules).
When the run completes results are published to a prometheus push-gateway so
they can displayed on the [traffic
generation](https://dashboards.gitlab.net/d/DK3VKVfiz/traffic-generation?orgId=1)
dashboard.
### Janitor
Once an hour in a [scheduled job](https://ops.gitlab.net/gitlab-com/gl-infra/traffic-generator/pipeline_schedules)
a [janitor script](https://ops.gitlab.net/gitlab-com/gl-infra/traffic-generator/blob/master/bin/janitor.bash)
is run that cleans up resources that GitLab-QA creates. This is so that we don't build up too many resources
under the gitlab-qa project.
## Adding or removing an HTTPS endpoint
Update the
......
general/images/deployer-example.png

53.9 KiB

# How to recover from a canceled coordinated pipeline?
> NOTE: This should be resolved as of January 2022: <https://gitlab.com/gitlab-org/gitlab/-/issues/341561>
With the current GitLab stable version - 14.3 at the moment of this guide - there's no
straightforward way to restart a canceled coordinated pipeline. Canceling a multi-project pipeline,
such as the coordinated pipeline, causes the [bridge jobs not to be initiated], recovering from this
......@@ -11,7 +13,7 @@ situation is complicated and time consuming. Because of this, **it's strongly ad
In the strange case the Release Manager requires to recover the coordinated pipeline, there are
some steps they can follow depending on the scenario.
### Continue with the deployments of the canceled pipelnie
### Continue with the deployments of the canceled pipeline
Bridge jobs, or deployment jobs, of a canceled coordinated pipeline are mark as `skipped`.
To initiate them Release Managers need to retry the previous job:
......
# Overview
This document describes the procedure to block Auto-Deploy operations
This document describes the procedure to lock Auto-Deploy operations
on an older branch.
# Blocking Auto-Deploy on a specific branch
## Locking auto-deploy to a specific branch
In a highly unstable production environment, a release manager may
decide to stick with an old auto-deploy branch. This option is
......@@ -53,17 +53,6 @@ If the instability is such that the release managers want to keep
maximum control over the content of the builds, consider disabling
[`auto_deploy:pick` schedule][auto_deploy:pick].
#### Changing default deployment targets
When a new auto-deploy package is ready, Omnibus will trigger a new
deployment. By default, the deployment triggers multiple environments
(`gstg,gprd-CNY,gprd`).
It is possible to alter this behavior by editing [the
`AUTO_DEPLOY_ENVIRONMENT` variable][omnibus variables] on the dev
instance. This option should not be taken lightly, as it's vital to
validate packages on staging before a production rollout.
### Cleanup
When the cause of the instability has been fixed, release managers
......@@ -72,9 +61,6 @@ should resume normal operations.
1. Activate [`auto_deploy:prepare` schedule][auto_deploy:prepare].
1. Run [`auto_deploy:prepare` schedule][auto_deploy:prepare].
1. **Optional** activate [`auto_deploy:pick` schedule][auto_deploy:pick].
1. **Optional** restore [the `AUTO_DEPLOY_ENVIRONMENT`
variable][omnibus variables] on the dev instance to
`gstg,gprd-cny,gprd`.
[auto_deploy:prepare]: https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/73/edit
[auto_deploy:pick]: https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/72/edit
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment