diff --git a/general/building-packages.md b/general/building-packages.md index 5f6c3f262d8f789a5f1ff861a48580e059f3c4e0..fd188da25ee969fdce50f81edb3f15e1231d59b2 100644 --- a/general/building-packages.md +++ b/general/building-packages.md @@ -2,10 +2,12 @@ Building artifacts for public consumption is handled on our dev instances via [`omnibus-gitlab`](https://dev.gitlab.org/gitlab/omnibus-gitlab). All builds are initiated via some sort of trigger: -* Nightly pipelines are kicked off via a Cron Job +* Nightly pipelines are kicked off [via a Cron Job][nightlies] * Builds of tagged packages are automatically started when that tag is pushed into the repo +[nightlies]: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipeline_schedules + # Nightly Builds ## Pipeline diff --git a/general/deploy/auto-deploy-transition.md b/general/deploy/auto-deploy-transition.md deleted file mode 100644 index 788874288712a5dad4a5f1a7bedc834b8cb8384e..0000000000000000000000000000000000000000 --- a/general/deploy/auto-deploy-transition.md +++ /dev/null @@ -1,9 +0,0 @@ -# Overview - -The [auto-deploy process overview](auto-deploy.md) provides details on the -building pieces of the process, a schedule overview and answers a few frequently -asked questions. - -Previous contents of this document were used to transition from the monthly deploy cadence to the new deploy system. For the most current information, -checkout the [Releases Handbook page](https://about.gitlab.com/handbook/engineering/releases/). - diff --git a/general/deploy/gitlab-com-deployer.md b/general/deploy/gitlab-com-deployer.md index 3f3e5bd396f59d71d98249ebd5cdcfdc175cdefc..42723b9bf409c299f2f24d589a696256c49302e3 100644 --- a/general/deploy/gitlab-com-deployer.md +++ b/general/deploy/gitlab-com-deployer.md @@ -1,14 +1,14 @@ -## Overview +# Overview -GitLab.com deployments are initiated by GitLab Chatops, this triggers +GitLab.com deployments are initiated by GitLab Chatops, which triggers a new pipeline on the [Deployer](https://ops.gitlab.net/gitlab-com/gl-infra/deployer) project. -To see all available options for deploy run `/chatops run deploy --help`: +To see all available options for deploy run `/chatops run deploy --help` ## Creating a new deployment for _upgrading_ GitLab Deployments are initiated using GitLab Chatops. -For example, to initiate a deployment of 11.8.0 to **staging**: +For example, to initiate a deployment of 11.8.0 to **staging** (default): ``` /chatops run deploy 11.8.0.ee.0 @@ -17,7 +17,7 @@ For example, to initiate a deployment of 11.8.0 to **staging**: ## Override variables Variables can be set on the deployer pipeline to change its behavior or -the jobs that will be executed. See [variables.md](../../runbooks/variables.md) +the jobs that will be executed. See [variables.md](../../runbooks/variables.md). ### Bypassing Failures @@ -63,28 +63,13 @@ the forced failure on the prepare job by using the option This essentially sends the variable `PRECHECK_IGNORE_ERRORS` to the deployment pipeline. -### Creating a new deployment for _rolling back_ GitLab +### Creating a new deployment for _rolling back_ GitLab to an earlier version -**IMPORTANT: Before initiating a rollback ensure to follow the [appropriate runbook](../../runbooks/rollback-a-deployment.md), -the following paragraph covers how to start a manual rollback pipeline on the deployer project in case of a chatops failure.** +Rollbacks are covered in detail in the [rollback runbook](../../runbooks/rollback-a-deployment.md). -In case of need it's possible to start a deployer pipeline by initiating a new pipeline -on the [deployer project](https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/new) -**default branch** and setting the environment and version CI variables. +## CI/CD Pipeline for a Deployer -* `DEPLOY_ENVIRONMENT`: **required** (i.e.: `gstg`, `gprd`) -* `DEPLOY_VERSION`: **required** -* `DEPLOY_ROLLBACK`: **required for rollback**, set to `true` to initiate a rollback pipeline - -_Note that all pipelines initiated on branches will automatically set the -`--dry-run` flag which can be useful for testing._ - - - - -### CICD Pipeline for a Deployer - -#### Assets +### Assets Assets are either extracted from the assets docker image if it is available or pulled from the omnibus package. This is done with a [shell script](https://gitlab.com/gitlab-com/gl-infra/deploy-tooling/-/blob/master/bin/fetch-and-upload-assets) in the deploy pipeline at the same time when we run database migrations in a job called `<ENV>-assets. See [.gitlab-ci](https://gitlab.com/gitlab-com/gl-infra/deployer/-/blob/d0b25b4c783d6010a9d7d0ce64ae49cdda813bb6/.gitlab-ci.yml#L345-359). @@ -116,7 +101,7 @@ When the browser requests an asset under `/assets`, it will either be for `asset There is an outstanding issue to simplify this by removing the Fastly CDN https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11411 -#### Upgrade Pipeline (default) +#### Upgrade Pipeline The following stages are run for upgrade for a single environment upgrade: @@ -143,30 +128,6 @@ graph LR; f ==> g>gitlab-qa]; ``` -#### Upgrade Pipeline (multi-environment) - -Multiple environments can be passed into the deployer project to chain -deployments, this works by setting `DEPLOY_ENVIRONMENT` to a comma separated -list of environments. The following chained environments are supported: - -* `DEPLOY_ENVIRONMENT=gstg-cny,gstg,gprd-cny`: Staging canary -> Staging -> Production canary -* `DEPLOY_ENVIRONMENT=gstg-cny,gstg,gprd-cny,gprd`: Staging canary -> Staging -> Production canary -> Production -* `DEPLOY_ENVIRONMENT=gstg,gprd-cny`: Staging -> Production canary -* `DEPLOY_ENVIRONMENT=gstg,gprd-cny,gprd`: Staging -> Production canary -> Production - -A deployment pipeline from staging -> production as the following stages that -provide safety QA checks and a manual promotion to production: - -```mermaid -graph LR; - a>Staging deploy] ==> b>Staging QA smoke tests]; - b ==> c>Production canary deploy]; - c ==> d>Canary smoke tests]; - d ==> e>Manaul approval]; - e ==> f>Production Deploy]; -``` - - #### Change-lock periods for deployment Before deploying to stage that receives production traffic there is an additional diff --git a/general/deploy/traffic-generation.md b/general/deploy/traffic-generation.md index c0bf34596f44c9edbb32bc7da938f49ac7f8de69..eab9b1e416248aab526e7ad66e96b3e367a9148e 100644 --- a/general/deploy/traffic-generation.md +++ b/general/deploy/traffic-generation.md @@ -27,22 +27,6 @@ Traffic is divided by the following backends: * pages: anon requests to a pages site * https_git: anon https request to git-upload-pack across various repositories. -### GitLab-QA - -GitLab-QA is configured to run every 10 minutes in a [scheduled -job](https://ops.gitlab.net/gitlab-com/gl-infra/traffic-generator/pipeline_schedules). -When the run completes results are published to a prometheus push-gateway so -they can displayed on the [traffic -generation](https://dashboards.gitlab.net/d/DK3VKVfiz/traffic-generation?orgId=1) -dashboard. - -### Janitor - -Once an hour in a [scheduled job](https://ops.gitlab.net/gitlab-com/gl-infra/traffic-generator/pipeline_schedules) -a [janitor script](https://ops.gitlab.net/gitlab-com/gl-infra/traffic-generator/blob/master/bin/janitor.bash) -is run that cleans up resources that GitLab-QA creates. This is so that we don't build up too many resources -under the gitlab-qa project. - ## Adding or removing an HTTPS endpoint Update the diff --git a/general/images/deployer-example.png b/general/images/deployer-example.png deleted file mode 100644 index 43eeb783eb63383d2a3fe42f4d947a79e96b2d51..0000000000000000000000000000000000000000 Binary files a/general/images/deployer-example.png and /dev/null differ diff --git a/runbooks/how-to-recover-from-canceled-pipeline.md b/runbooks/how-to-recover-from-canceled-pipeline.md index da9413a6b99c9ddb053bf7f85c3a53db88e1402d..75551ecfa9c51c27c37e79a4be952617dfbec383 100644 --- a/runbooks/how-to-recover-from-canceled-pipeline.md +++ b/runbooks/how-to-recover-from-canceled-pipeline.md @@ -1,5 +1,7 @@ # How to recover from a canceled coordinated pipeline? +> NOTE: This should be resolved as of January 2022: <https://gitlab.com/gitlab-org/gitlab/-/issues/341561> + With the current GitLab stable version - 14.3 at the moment of this guide - there's no straightforward way to restart a canceled coordinated pipeline. Canceling a multi-project pipeline, such as the coordinated pipeline, causes the [bridge jobs not to be initiated], recovering from this @@ -11,7 +13,7 @@ situation is complicated and time consuming. Because of this, **it's strongly ad In the strange case the Release Manager requires to recover the coordinated pipeline, there are some steps they can follow depending on the scenario. -### Continue with the deployments of the canceled pipelnie +### Continue with the deployments of the canceled pipeline Bridge jobs, or deployment jobs, of a canceled coordinated pipeline are mark as `skipped`. To initiate them Release Managers need to retry the previous job: diff --git a/runbooks/tagging-past-auto-deploy-branches.md b/runbooks/tagging-past-auto-deploy-branches.md index 086d54990598273aa56538e312ab7f052caf92a0..ed5cdffed512730fddb4c18163c0f27cfcb00c29 100644 --- a/runbooks/tagging-past-auto-deploy-branches.md +++ b/runbooks/tagging-past-auto-deploy-branches.md @@ -1,9 +1,9 @@ # Overview -This document describes the procedure to block Auto-Deploy operations +This document describes the procedure to lock Auto-Deploy operations on an older branch. -# Blocking Auto-Deploy on a specific branch +## Locking auto-deploy to a specific branch In a highly unstable production environment, a release manager may decide to stick with an old auto-deploy branch. This option is @@ -53,17 +53,6 @@ If the instability is such that the release managers want to keep maximum control over the content of the builds, consider disabling [`auto_deploy:pick` schedule][auto_deploy:pick]. -#### Changing default deployment targets - -When a new auto-deploy package is ready, Omnibus will trigger a new -deployment. By default, the deployment triggers multiple environments -(`gstg,gprd-CNY,gprd`). - -It is possible to alter this behavior by editing [the -`AUTO_DEPLOY_ENVIRONMENT` variable][omnibus variables] on the dev -instance. This option should not be taken lightly, as it's vital to -validate packages on staging before a production rollout. - ### Cleanup When the cause of the instability has been fixed, release managers @@ -72,9 +61,6 @@ should resume normal operations. 1. Activate [`auto_deploy:prepare` schedule][auto_deploy:prepare]. 1. Run [`auto_deploy:prepare` schedule][auto_deploy:prepare]. 1. **Optional** activate [`auto_deploy:pick` schedule][auto_deploy:pick]. -1. **Optional** restore [the `AUTO_DEPLOY_ENVIRONMENT` -variable][omnibus variables] on the dev instance to -`gstg,gprd-cny,gprd`. [auto_deploy:prepare]: https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/73/edit [auto_deploy:pick]: https://ops.gitlab.net/gitlab-org/release/tools/-/pipeline_schedules/72/edit