README.md 41 KB
Newer Older
1 2 3 4 5 6
---
stage: Enablement
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---

Eldridge Henley's avatar
Eldridge Henley committed
7
# Update GitLab installed with the Omnibus GitLab package
8

Eldridge Henley's avatar
Eldridge Henley committed
9
Before following these instructions, note the following:
10

Eldridge Henley's avatar
Eldridge Henley committed
11 12 13 14 15 16 17 18 19 20 21 22
- [Upgrade recommendations](https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations)
  has suggestions on when to upgrade.
- If you are upgrading from a non-Omnibus installation to an Omnibus installation, see
  [Upgrading from a non-Omnibus installation to an Omnibus installation](convert_to_omnibus.md).

CAUTION: **Caution:**
If you aren't [using the current major version](#mandatory-upgrade-paths-for-version-upgrades),
you **must** follow the
[upgrade recommendations](https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations)
when updating to the current version.

## Version-specific changes
23

24 25 26
It's important to ensure that any background migrations have been fully completed
before upgrading to a new major version. To see the current size of the `background_migration` queue,
[check for background migrations before upgrading](https://docs.gitlab.com/ee/update/README.html#checking-for-background-migrations-before-upgrading).
Eldridge Henley's avatar
Eldridge Henley committed
27 28 29 30 31
We recommend performing upgrades between major and minor releases no more than once per
week, to allow time for background migrations to finish. Decrease the time required to
complete these migrations by increasing the number of
[Sidekiq workers](https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html)
that can process jobs in the `background_migration` queue.
32

Evan Read's avatar
Evan Read committed
33
Updating to major versions might need some manual intervention. For more information,
34 35
check the version your are updating to:

36
- [GitLab 13](gitlab_13_changes.md)
37
- [GitLab 12](gitlab_12_changes.md)
38
- [GitLab 11](gitlab_11_changes.md)
39 40 41 42
- [GitLab 10](gitlab_10_changes.md)
- [GitLab 8](gitlab_8_changes.md)
- [GitLab 7](gitlab_7_changes.md)
- [GitLab 6](gitlab_6_changes.md)
43

44 45
## Mandatory upgrade paths for version upgrades

Evan Read's avatar
Evan Read committed
46
From GitLab 10.8, upgrade paths are enforced for version upgrades by
47
default. This restricts performing direct upgrades that skip major versions (for
Eldridge Henley's avatar
Eldridge Henley committed
48 49
example 10.3 to 12.7 in one jump) that **can break GitLab
installations** due to multiple reasons like deprecated or removed configuration
50 51 52 53
settings, upgrade of internal tools and libraries etc. Users will have to follow
the [official upgrade recommendations](https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations)
while upgrading their GitLab instances.

54
## Updating methods
55

56 57
There are two ways to update Omnibus GitLab:

Evan Read's avatar
Evan Read committed
58 59
- [Using the official repositories](#update-using-the-official-repositories).
- [Using a manually-downloaded package](#update-using-a-manually-downloaded-package).
60

61 62 63
Both will automatically back up the GitLab database before installing a newer
GitLab version. You may skip this automatic backup by creating an empty file
at `/etc/gitlab/skip-auto-backup`:
64

Evan Read's avatar
Evan Read committed
65
```shell
66 67 68
sudo touch /etc/gitlab/skip-auto-backup
```

69
NOTE: **Note:**
70 71 72 73
For safety reasons, you should maintain an up-to-date backup on your own if you plan to use this flag.

NOTE: **Note**
When upgrading to a new major version, remember to first [check for background migrations](https://docs.gitlab.com/ee/update/README.html#checking-for-background-migrations-before-upgrading).
74

75
NOTE: **Note**
76
Unless you are following the steps in [Zero downtime updates](#zero-downtime-updates), your GitLab application will not be available to users while an update is in progress. They will either see a "Deploy in progress" message or a "502" error in their web browser.
77

Evan Read's avatar
Evan Read committed
78
### Update using the official repositories
79

Marcel Amirault's avatar
Marcel Amirault committed
80 81
If you have installed Omnibus GitLab [Community Edition](https://about.gitlab.com/install/?version=ce)
or [Enterprise Edition](https://about.gitlab.com/install/), then the
82
official GitLab repository should have already been set up for you.
83

Eldridge Henley's avatar
Eldridge Henley committed
84
To update to a newer GitLab version, run:
85

Eldridge Henley's avatar
Eldridge Henley committed
86
- For GitLab Community Edition:
87

Eldridge Henley's avatar
Eldridge Henley committed
88 89 90 91
  ```shell
  # Debian/Ubuntu
  sudo apt-get update
  sudo apt-get install gitlab-ce
92

Eldridge Henley's avatar
Eldridge Henley committed
93 94 95 96
  # Centos/RHEL
  sudo yum install gitlab-ce
  ```

Evan Read's avatar
Evan Read committed
97
- For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/):
Eldridge Henley's avatar
Eldridge Henley committed
98 99 100 101 102 103 104 105 106

  ```shell
  # Debian/Ubuntu
  sudo apt-get update
  sudo apt-get install gitlab-ee

  # Centos/RHEL
  sudo yum install gitlab-ee
  ```
107

Evan Read's avatar
Evan Read committed
108
### Update using a manually-downloaded package
109

Evan Read's avatar
Evan Read committed
110
If for some reason you don't use the official repositories, you can
111
[download the package and install it manually](../manual_install.md).
112

Evan Read's avatar
Evan Read committed
113
## Update Community Edition to Enterprise Edition
Eldridge Henley's avatar
Eldridge Henley committed
114

Evan Read's avatar
Evan Read committed
115 116 117
To upgrade an existing GitLab Community Edition (CE) server installed using the Omnibus GitLab
packages to GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) (EE), you install the EE
package on top of CE.
Eldridge Henley's avatar
Eldridge Henley committed
118

Evan Read's avatar
Evan Read committed
119 120 121
Upgrading from the same version of CE to EE is not explicitly necessary, and any standard upgrade
(for example, CE 12.0 to EE 12.1) should work. However, in the following steps we assume that
you are upgrading the same version (for example, CE 12.1 to EE 12.1), which is **recommended**.
Eldridge Henley's avatar
Eldridge Henley committed
122 123

CAUTION: **Caution:**
Evan Read's avatar
Evan Read committed
124 125 126 127
When updating to EE from CE, avoid reverting back to CE if you plan on going to EE again in the
future. Reverting back to CE can cause
[database issues](#500-error-when-accessing-project--settings--repository-on-omnibus-installs)
that may require Support intervention.
128 129 130

The steps can be summed up to:

131 132
1. Find the currently installed GitLab version:

133
   **For Debian/Ubuntu**
134

Evan Read's avatar
Evan Read committed
135
   ```shell
136 137
   sudo apt-cache policy gitlab-ce | grep Installed
   ```
138

Eldridge Henley's avatar
Eldridge Henley committed
139 140
   The output should be similar to: `Installed: 13.0.4-ce.0`. In that case,
   the equivalent Enterprise Edition version will be: `13.0.4-ee.0`. Write this
141
   value down.
142

143
   **For CentOS/RHEL**
144

Evan Read's avatar
Evan Read committed
145
   ```shell
146 147
   sudo rpm -q gitlab-ce
   ```
148

Eldridge Henley's avatar
Eldridge Henley committed
149
   The output should be similar to: `gitlab-ce-13.0.4-ce.0.el8.x86_64`. In that
150
   case, the equivalent Enterprise Edition version will be:
Eldridge Henley's avatar
Eldridge Henley committed
151
   `gitlab-ee-13.0.4-ee.0.el8.x86_64`. Write this value down.
152

153 154
1. Add the `gitlab-ee` [Apt or Yum repository](https://packages.gitlab.com/gitlab/gitlab-ee/install):

155
   **For Debian/Ubuntu**
156

Evan Read's avatar
Evan Read committed
157
   ```shell
158 159
   curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
   ```
160

161
   **For CentOS/RHEL**
162

Evan Read's avatar
Evan Read committed
163
   ```shell
164 165
   curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | sudo bash
   ```
166

167 168 169 170
   The above command will find your OS version and automatically set up the
   repository. If you are not comfortable installing the repository through a
   piped script, you can first
   [check its contents](https://packages.gitlab.com/gitlab/gitlab-ee/install).
171 172

1. Next, install the `gitlab-ee` package. Note that this will automatically
173
   uninstall the `gitlab-ce` package on your GitLab server. `reconfigure`
Eldridge Henley's avatar
Eldridge Henley committed
174 175
   Omnibus right after the `gitlab-ee` package is installed. **Make sure that you
   install the exact same GitLab version**:
176

177
   **For Debian/Ubuntu**
178

Evan Read's avatar
Evan Read committed
179
   ```shell
180 181
   ## Make sure the repositories are up-to-date
   sudo apt-get update
182

183
   ## Install the package using the version you wrote down from step 1
Eldridge Henley's avatar
Eldridge Henley committed
184
   sudo apt-get install gitlab-ee=13.0.4-ee.0
185

186 187 188
   ## Reconfigure GitLab
   sudo gitlab-ctl reconfigure
   ```
189

190
   **For CentOS/RHEL**
191

Evan Read's avatar
Evan Read committed
192
   ```shell
193
   ## Install the package using the version you wrote down from step 1
Eldridge Henley's avatar
Eldridge Henley committed
194
   sudo yum install gitlab-ee-13.0.4-ee.0.el8.x86_64
195

196 197 198
   ## Reconfigure GitLab
   sudo gitlab-ctl reconfigure
   ```
199

Evan Read's avatar
Evan Read committed
200
   NOTE: **Note:**
Eldridge Henley's avatar
Eldridge Henley committed
201 202 203 204
   To upgrade to EE and also update GitLab to the latest version at the same time, you
   can omit version information from the commands above. That is, run
   `sudo apt-get install gitlab-ee` or `sudo yum install gitlab-ee` depending on your
   distribution.
205

206 207 208
1. Now go to the GitLab admin panel of your server (`/admin/license/new`) and
   upload your license file.

209 210 211
1. After you confirm that GitLab is working as expected, you may remove the old
   Community Edition repository:

212
   **For Debian/Ubuntu**
213

Evan Read's avatar
Evan Read committed
214
   ```shell
215 216
   sudo rm /etc/apt/sources.list.d/gitlab_gitlab-ce.list
   ```
217

218
   **For CentOS/RHEL**
219

Evan Read's avatar
Evan Read committed
220
   ```shell
221 222
   sudo rm /etc/yum.repos.d/gitlab_gitlab-ce.repo
   ```
223 224

That's it! You can now use GitLab Enterprise Edition! To update to a newer
Evan Read's avatar
Evan Read committed
225
version, follow [Update using the official repositories](#update-using-the-official-repositories).
226

227
NOTE: **Note:**
228
If you want to use `dpkg`/`rpm` instead of `apt-get`/`yum`, go through the first
Evan Read's avatar
Evan Read committed
229 230
step to find the current GitLab version and then follow
[Update using a manually-downloaded package](#update-using-a-manually-downloaded-package).
231

232
## Zero downtime updates
233

234
NOTE: **Note:**
235
This is only available in GitLab 9.1.0 or newer. Skipping restarts during `reconfigure` with `/etc/gitlab/skip-auto-reconfigure` was added in [version 10.6](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2270). If running a version prior to 10.6, you will need to create `/etc/gitlab/skip-auto-migrations`.
236

237 238
It's possible to upgrade to a newer version of GitLab without having to take
your GitLab instance offline.
239

240 241
Verify that you can upgrade with no downtime by checking the
[Upgrading without downtime section](https://docs.gitlab.com/ee/update/README.html#upgrading-without-downtime) of the update document.
242

243 244
If you meet all the requirements above, follow these instructions in order. There are three sets of steps, depending on your deployment type:

Eldridge Henley's avatar
Eldridge Henley committed
245 246 247 248
| Deployment type                                                 | Description                                       |
| --------------------------------------------------------------- | ------------------------------------------------  |
| [Single-node](#single-node-deployment)                          | GitLab CE/EE on a single node                     |
| [Gitaly Cluster](#gitaly-cluster)                               | GitLab CE/EE using HA architecture for Gitaly Cluster             |
Evan Read's avatar
Evan Read committed
249 250
| [Multi-node / PostgreSQL HA](#use-postgresql-ha)                | GitLab CE/EE using HA architecture for PostgreSQL |
| [Multi-node / Redis HA](#use-redis-ha-using-sentinel-premium-only)           | GitLab CE/EE using HA architecture for Redis |
251 252
| [Geo](#geo-deployment-premium-only)                                          | GitLab EE with Geo enabled                        |
| [Multi-node / HA with Geo](#multi-node--ha-deployment-with-geo-premium-only) | GitLab CE/EE on multiple nodes                    |
253

254 255 256 257 258
Each type of deployment will require that you hot reload the `puma` (or `unicorn`) and `sidekiq` processes on all nodes running these
services after you've upgraded. The reason for this is that those processes each load the GitLab Rails application which reads and loads
the database schema into memory when starting up. Each of these processes will need to be reloaded (or restarted in the case of `sidekiq`)
to re-read any database changes that have been made by post-deployment migrations.

259 260
### Single-node deployment

Eldridge Henley's avatar
Eldridge Henley committed
261
Before following these instructions, note the following **important** information:
262

Eldridge Henley's avatar
Eldridge Henley committed
263 264 265 266 267 268 269 270 271 272 273
- On single-node Omnibus deployments, zero down-time updates are not possible when
  using Puma because Puma always requires a complete restart (Puma replaced Unicorn as
  the default in GitLab 13.0 unless
  [specifically disabled](../settings/unicorn.md#enabling-unicorn)). This is because the
  [phased restart](https://github.com/puma/puma/blob/master/README.md#clustered-mode)
  feature of Puma does not work with the way it is configured in GitLab's all-in-one
  packages (cluster-mode with app preloading).
- While it is possible to minimize downtime on a single-node instance by following
  these instructions, **it is not possible to always achieve true zero downtime
  updates**. Users may see some connections timeout or be refused for a few minutes,
  depending on which services need to restart.
274 275 276 277 278

1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. During software
   installation only, this will prevent the upgrade from running
   `gitlab-ctl reconfigure` and automatically running database migrations

Evan Read's avatar
Evan Read committed
279
   ```shell
280 281
   sudo touch /etc/gitlab/skip-auto-reconfigure
   ```
282

Eldridge Henley's avatar
Eldridge Henley committed
283
1. Update the GitLab package:
284

Eldridge Henley's avatar
Eldridge Henley committed
285
   - For GitLab Community Edition:
286

Eldridge Henley's avatar
Eldridge Henley committed
287 288 289 290 291 292 293 294 295
     ```shell
     # Debian/Ubuntu
     sudo apt-get update
     sudo apt-get install gitlab-ce

     # Centos/RHEL
     sudo yum install gitlab-ce
     ```

Evan Read's avatar
Evan Read committed
296
   - For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/):
Eldridge Henley's avatar
Eldridge Henley committed
297 298 299 300 301 302 303 304 305

     ```shell
     # Debian/Ubuntu
     sudo apt-get update
     sudo apt-get install gitlab-ee

     # Centos/RHEL
     sudo yum install gitlab-ee
     ```
306

Evan Read's avatar
Evan Read committed
307
   NOTE: **Note**
Eldridge Henley's avatar
Eldridge Henley committed
308 309
   The above commands use the latest version of GitLab. Use a version-specific package
   to update to an older version.
310

311
1. To get the regular migrations and latest code in place, run
312

Evan Read's avatar
Evan Read committed
313
   ```shell
314
   sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure
315
   ```
316

317
1. Once the node is updated and `reconfigure` finished successfully, complete the migrations with
318

Evan Read's avatar
Evan Read committed
319
   ```shell
320 321
   sudo gitlab-rake db:migrate
   ```
322

323
1. Hot reload `unicorn` (or `puma`) and `sidekiq` services
324

Evan Read's avatar
Evan Read committed
325
   ```shell
326
   sudo gitlab-ctl hup unicorn
327
   sudo gitlab-ctl restart sidekiq
328
   ```
329

330 331 332 333 334
NOTE: **Note:**
If you do not want to run zero downtime upgrades in the future, make
sure you remove `/etc/gitlab/skip-auto-reconfigure` after
you've completed these steps.

335
### Multi-node / HA deployment
336

Evan Read's avatar
Evan Read committed
337
#### Use a load balancer in front of web (Puma/Unicorn) nodes
338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421

With Puma, single node zero-downtime updates are no longer possible. To achieve
HA with zero-downtime updates, at least two nodes are required to be used with a
load balancer which distributes the connections properly across both nodes.

The load balancer in front of the application nodes must be configured to check
proper health check endpoints to check if the service is accepting traffic or
not. For Puma and Unicorn, the `/-/readiness` endpoint should be used, while
`/readiness` endpoint can be used for Sidekiq and other services.

Upgrades on web (Puma/Unicorn) nodes must be done in a rolling manner, one after
another, ensuring at least one node is always up to serve traffic. This is
required to ensure zero-downtime.

Both Puma and Unicorn will enter a blackout period as part of the upgrade,
during which they continue to accept connections but will mark their respective
health check endpoints to be unhealthy. On seeing this, the load balancer should
disconnect them gracefully.

Both Puma and Unicorn will restart only after completing all the currently
processing requests. This ensures data and service integrity. Once they have
restarted, the health check end points will be marked healthy.

The nodes must be updated in the following order to update an HA instance using
load balancer to latest GitLab version.

1. Select one application node as a deploy node and complete the following steps
   on it:

    1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This will
       prevent the upgrade from running `gitlab-ctl reconfigure` and
       automatically running database migrations:

        ```shell
        sudo touch /etc/gitlab/skip-auto-reconfigure
        ```

    1. Update the GitLab package:

       ```shell
       # Debian/Ubuntu
       sudo apt-get update && sudo apt-get install gitlab-ce

       # Centos/RHEL
       sudo yum install gitlab-ce
       ```

       If you are an Enterprise Edition user, replace `gitlab-ce` with
       `gitlab-ee` in the above command.

    1. Get the regular migrations and latest code in place:

       ```shell
       sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure
       ```

    1. Ensure services use the latest code:

       ```shell
       sudo gitlab-ctl hup puma
       sudo gitlab-ctl restart sidekiq
       ```

1. Complete the following steps on the other Puma/Unicorn/Sidekiq nodes, one
   after another. Always ensure at least one of such nodes is up and running,
   and connected to the load balancer before proceeding to the next node.

    1. Update the GitLab package and ensure a `reconfigure` is run as part of
       it. If not (due to `/etc/gitlab/skip-auto-reconfigure` file being
       present), run `sudo gitlab-ctl reconfigure` manually.

    1. Ensure services use latest code:

       ```shell
       sudo gitlab-ctl hup puma
       sudo gitlab-ctl restart sidekiq
       ```

1. On the deploy node, run the post-deployment migrations:

      ```shell
      sudo gitlab-rake db:migrate
      ```

422
#### Gitaly Cluster
423

424 425 426
[Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/praefect.html) is built using
Gitaly and the Praefect component. It has its own PostgreSQL database, independent of the rest of
the application.
427 428 429 430 431 432 433 434 435 436 437 438

Before you update the main application you need to update Praefect.
Out of your Praefect nodes, pick one to be your Praefect deploy node.
This is where you will install the new Omnibus package first and run
database migrations.

**Praefect deploy node**

- Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. During software
  installation only, this will prevent the upgrade from running
  `gitlab-ctl reconfigure` and restarting GitLab before database migrations have been applied:

Evan Read's avatar
Evan Read committed
439
  ```shell
440 441 442 443 444 445 446 447 448 449 450 451 452
  sudo touch /etc/gitlab/skip-auto-reconfigure
  ```

- Ensure that `praefect['auto_migrate'] = true` is set in `/etc/gitlab/gitlab.rb`

**All other Praefect nodes (not the Praefect deploy node)**

- Ensure that `praefect['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`

**Praefect deploy node**

- Update the GitLab package:

Evan Read's avatar
Evan Read committed
453
  ```shell
454 455 456 457 458 459 460 461 462 463 464
  # Debian/Ubuntu
  sudo apt-get update && sudo apt-get install gitlab-ce

  # Centos/RHEL
  sudo yum install gitlab-ce
  ```

  If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command.

- To apply the Praefect database migrations and restart Praefect, run:

Evan Read's avatar
Evan Read committed
465
  ```shell
466 467 468 469 470 471 472
  sudo gitlab-ctl reconfigure
  ```

**All other Praefect nodes (not the Praefect deploy node)**

- Update the GitLab package:

Evan Read's avatar
Evan Read committed
473
  ```shell
474 475 476 477 478 479 480
  sudo apt-get update && sudo apt-get install gitlab-ce
  ```

  If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command.

- Ensure nodes are running the latest code:

Evan Read's avatar
Evan Read committed
481
  ```shell
482 483 484
  sudo gitlab-ctl reconfigure
  ```

Evan Read's avatar
Evan Read committed
485
#### Use PostgreSQL HA
486

Evan Read's avatar
Evan Read committed
487
Pick a node to be the `Deploy Node`. It can be any node, but it must be the same
488 489 490 491
node throughout the process.

**Deploy node**

492
- Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. During software
493
  installation only, this will prevent the upgrade from running
494
  `gitlab-ctl reconfigure` and restarting GitLab before database migrations have been applied.
495

Evan Read's avatar
Evan Read committed
496
  ```shell
497 498
  sudo touch /etc/gitlab/skip-auto-reconfigure
  ```
499

500
**All nodes (including the Deploy node)**
501

502
- Ensure that `gitlab_rails['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`
503

504 505 506 507
**Gitaly only nodes**

- Update the GitLab package

Evan Read's avatar
Evan Read committed
508
  ```shell
509 510 511 512 513 514 515 516 517 518 519
  # Debian/Ubuntu
  sudo apt-get update && sudo apt-get install gitlab-ce

  # Centos/RHEL
  sudo yum install gitlab-ce
  ```

  If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command.

- Ensure nodes are running the latest code

Evan Read's avatar
Evan Read committed
520
  ```shell
521 522 523
  sudo gitlab-ctl reconfigure
  ```

524 525
**Deploy node**

526
- Update the GitLab package
527

Evan Read's avatar
Evan Read committed
528
  ```shell
529 530
  # Debian/Ubuntu
  sudo apt-get update && sudo apt-get install gitlab-ce
531

532 533 534
  # Centos/RHEL
  sudo yum install gitlab-ce
  ```
535

536
  If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command.
537

538 539 540 541
- If you're using PgBouncer:

  You'll need to bypass PgBouncer and connect directly to the database master
  before running migrations.
Michael Kozono's avatar
Michael Kozono committed
542

543 544 545 546 547
  Rails uses an advisory lock when attempting to run a migration to prevent
  concurrent migrations from running on the same database. These locks are
  not shared across transactions, resulting in `ActiveRecord::ConcurrentMigrationError`
  and other issues when running database migrations using PgBouncer in transaction
  pooling mode.
Michael Kozono's avatar
Michael Kozono committed
548

549 550
  To find the master node, run the following on a database node:

Evan Read's avatar
Evan Read committed
551
  ```shell
552 553 554 555 556 557 558
  sudo gitlab-ctl repmgr cluster show
  ```

  Then, in your `gitlab.rb` file on the deploy node, update
  `gitlab_rails['db_host']` and `gitlab_rails['db_port']` with the database
  master's host and port.

559
- To get the regular database migrations and latest code in place, run
560

Evan Read's avatar
Evan Read committed
561
  ```shell
562
  sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure
563
  ```
564 565 566

**All other nodes (not the Deploy node)**

567
- Update the GitLab package
568

Evan Read's avatar
Evan Read committed
569
  ```shell
570 571
  sudo apt-get update && sudo apt-get install gitlab-ce
  ```
572

573
  If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command.
574

575
- Ensure nodes are running the latest code
576

Evan Read's avatar
Evan Read committed
577
  ```shell
578 579
  sudo gitlab-ctl reconfigure
  ```
580

581 582 583 584
**Deploy node**

- Run post-deployment database migrations on deploy node to complete the migrations with

Evan Read's avatar
Evan Read committed
585
  ```shell
586 587 588
  sudo gitlab-rake db:migrate
  ```

589
**For nodes that run Puma/Unicorn or Sidekiq**
590

591
- Hot reload `puma` (or `unicorn`) and `sidekiq` services
592

Evan Read's avatar
Evan Read committed
593
  ```shell
594
  sudo gitlab-ctl hup puma
595
  sudo gitlab-ctl restart sidekiq
596
  ```
597

598 599 600 601
- If you're using PgBouncer:

  Change your `gitlab.rb` to point back to PgBouncer and run:

Evan Read's avatar
Evan Read committed
602
  ```shell
603 604 605
  sudo gitlab-ctl reconfigure
  ```

606 607 608 609 610 611
NOTE: **Note:**
If you do not want to run zero downtime upgrades in the future, make
sure you remove `/etc/gitlab/skip-auto-reconfigure` and revert
setting `gitlab_rails['auto_migrate'] = false` in
`/etc/gitlab/gitlab.rb` after you've completed these steps.

Evan Read's avatar
Evan Read committed
612
#### Use Redis HA (using Sentinel) **(PREMIUM ONLY)**
613 614 615 616 617 618 619 620 621 622 623

Package upgrades may involve version updates to the bundled Redis service. On
instances using [Redis HA](https://docs.gitlab.com/ee/administration/high_availability/redis.html),
upgrades must follow a proper order to ensure minimum downtime, as specified
below. This doc assumes the official guides are being followed to setup Redis
HA.

##### In the application node

According to [official Redis docs](https://redis.io/topics/admin#upgrading-or-restarting-a-redis-instance-without-downtime),
the easiest way to update an HA instance using Sentinel is to upgrade the
Eldridge Henley's avatar
Eldridge Henley committed
624
secondaries one after the other, perform a manual failover from current
625 626 627 628 629 630 631
primary (running old version) to a recently upgraded secondary (running a new
version), and then upgrade the original primary. For this, we need to know
the address of the current Redis primary.

- If your application node is running GitLab 12.7.0 or later, you can use the
following command to get address of current Redis primary

632
  ```shell
633 634 635 636 637 638 639
  sudo gitlab-ctl get-redis-master
  ```

- If your application node is running a version older than GitLab 12.7.0, you
  will have to run the underlying `redis-cli` command (which `get-redis-master`
  command uses) to fetch information about the primary.

640 641
  1. Get the address of one of the sentinel nodes specified as
     `gitlab_rails['redis_sentinels']` in `/etc/gitlab/gitlab.rb`
642

643 644
  1. Get the Redis master name specified as `redis['master_name']` in
     `/etc/gitlab/gitlab.rb`
645

646
  1. Run the following command
647

648 649 650
     ```shell
     sudo /opt/gitlab/embedded/bin/redis-cli -h <sentinel host> -p <sentinel port> SENTINEL get-master-addr-by-name <redis master name>
     ```
651 652 653 654 655 656 657 658 659 660 661

##### In the Redis secondary nodes

1. Install package for new version.

1. Run `sudo gitlab-ctl reconfigure`, if a reconfigure is not run as part of
   installation (due to `/etc/gitlab/skip-auto-reconfigure` file being present).

1. If reconfigure warns about a pending Redis/Sentinel restart, restart the
   corresponding service

662
   ```shell
663 664 665 666 667 668 669 670 671 672 673 674 675
   sudo gitlab-ctl restart redis
   sudo gitlab-ctl restart sentinel
   ```

##### In the Redis primary node

Before upgrading the Redis primary node, we need to perform a failover so that
one of the recently upgraded secondary nodes becomes the new primary. Once the
failover is complete, we can go ahead and upgrade the original primary node.

1. Stop Redis service in Redis primary node so that it fails over to a secondary
   node

676
   ```shell
677 678 679 680 681 682 683 684 685 686
   sudo gitlab-ctl stop redis
   ```

1. Wait for failover to be complete. You can verify it by periodically checking
   details of the current Redis primary node (as mentioned above). If it starts
   reporting a new IP, failover is complete.

1. Start Redis again in that node, so that it starts following the current
   primary node.

687
   ```shell
688 689 690 691 692 693 694 695 696 697 698
   sudo gitlab-ctl start redis
   ```

1. Install package corresponding to new version.

1. Run `sudo gitlab-ctl reconfigure`, if a reconfigure is not run as part of
   installation (due to `/etc/gitlab/skip-auto-reconfigure` file being present).

1. If reconfigure warns about a pending Redis/Sentinel restart, restart the
   corresponding service

699
   ```shell
700 701 702 703 704 705 706 707 708
   sudo gitlab-ctl restart redis
   sudo gitlab-ctl restart sentinel
   ```

##### Update the application node

Install the package for new version and follow regular package upgrade
procedure.

709
### Geo deployment **(PREMIUM ONLY)**
710

711 712
NOTE: **Note:**
The order of steps is important. While following these steps, make
713
sure you follow them in the right order, on the correct node.
714

715
Log in to your **primary** node, executing the following:
716 717 718 719 720

1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. During software
   installation only, this will prevent the upgrade from running
   `gitlab-ctl reconfigure` and automatically running database migrations

Evan Read's avatar
Evan Read committed
721
   ```shell
722 723
   sudo touch /etc/gitlab/skip-auto-reconfigure
   ```
724 725 726

1. Update the GitLab package

Evan Read's avatar
Evan Read committed
727
   ```shell
728 729
   # Debian/Ubuntu
   sudo apt-get update && sudo apt-get install gitlab-ee
730

731 732 733
   # Centos/RHEL
   sudo yum install gitlab-ee
   ```
734

735
1. To get the database migrations and latest code in place, run
736

Evan Read's avatar
Evan Read committed
737
   ```shell
738
   sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure
739
   ```
740

741 742 743 744 745 746 747
NOTE: **Note:**
After this step you can get an outdated FDW remote schema on your
secondary nodes. While it is not important to worry about at this
point, you can check out the
[Geo troubleshooting documentation](https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html#geo-database-has-an-outdated-fdw-remote-schema-error)
to resolve this.

748
1. Hot reload `puma` (or `unicorn`) and `sidekiq` services
749

Evan Read's avatar
Evan Read committed
750
   ```shell
751
   sudo gitlab-ctl hup puma
752
   sudo gitlab-ctl restart sidekiq
753
   ```
754

755
On each **secondary** node, executing the following:
756 757 758 759 760

1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. During software
   installation only, this will prevent the upgrade from running
   `gitlab-ctl reconfigure` and automatically running database migrations

Evan Read's avatar
Evan Read committed
761
   ```shell
762 763
   sudo touch /etc/gitlab/skip-auto-reconfigure
   ```
764 765 766

1. Update the GitLab package

Evan Read's avatar
Evan Read committed
767
   ```shell
768 769
   # Debian/Ubuntu
   sudo apt-get update && sudo apt-get install gitlab-ee
770

771 772 773
   # Centos/RHEL
   sudo yum install gitlab-ee
   ```
774

775
1. To get the database migrations and latest code in place, run
776

Evan Read's avatar
Evan Read committed
777
   ```shell
778
   sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure
779
   ```
780

781
1. Hot reload `puma` (or `unicorn`), `sidekiq` and restart `geo-logcursor` services
782

Evan Read's avatar
Evan Read committed
783
   ```shell
784
   sudo gitlab-ctl hup puma
785
   sudo gitlab-ctl restart sidekiq
786 787
   sudo gitlab-ctl restart geo-logcursor
   ```
788

Fabian Zimmer's avatar
Fabian Zimmer committed
789 790
1. Run post-deployment database migrations, specific to the Geo database

Evan Read's avatar
Evan Read committed
791
   ```shell
Fabian Zimmer's avatar
Fabian Zimmer committed
792 793 794
   sudo gitlab-rake geo:db:migrate
   ```

795
After all **secondary** nodes are updated, finalize
796 797 798
the update on the **primary** node:

- Run post-deployment database migrations
799

Evan Read's avatar
Evan Read committed
800
   ```shell
801
   sudo gitlab-rake db:migrate
802
   ```
803

Michael Kozono's avatar
Michael Kozono committed
804 805 806 807 808 809 810
On each **secondary**, ensure the FDW tables are up-to-date.

1. Wait for the **primary** migrations to finish.

1. Wait for the **primary** migrations to replicate. You can find "Data
   replication lag" for each node listed on `Admin Area > Geo`.

811
After updating all nodes (both **primary** and all **secondaries**), check their status:
812 813

- Verify Geo configuration and dependencies
814

Evan Read's avatar
Evan Read committed
815
   ```shell
816
   sudo gitlab-rake gitlab:geo:check
817
   ```
818

819 820 821 822 823 824
NOTE: **Note:**
If you do not want to run zero downtime upgrades in the future, make
sure you remove `/etc/gitlab/skip-auto-reconfigure` and revert
setting `gitlab_rails['auto_migrate'] = false` in
`/etc/gitlab/gitlab.rb` after you've completed these steps.

825
### Multi-node / HA deployment with Geo **(PREMIUM ONLY)**
826

827 828 829 830
This section describes the steps required to upgrade a multi-node / HA
deployment with Geo. Some steps must be performed on a particular node. This
node will be known as the “deploy node” and is noted through the following
instructions.
831

832
Updates must be performed in the following order:
833

834
1. Update Geo **primary** multi-node deployment.
835
1. Update Geo **secondary** multi-node deployments.
836
1. Post-deployment migrations and checks.
837

838 839 840 841 842 843 844
#### Step 1: Choose a "deploy node" for each deployment

You now need to choose:

- One instance for use as the **primary** "deploy node" on the Geo **primary** multi-node deployment.
- One instance for use as the **secondary** "deploy node" on each Geo **secondary** multi-node deployment.

845
Deploy nodes must be configured to be running Puma/Unicorn or Sidekiq or the `geo-logcursor` daemon. In order
846 847
to avoid any downtime, they must not be in use during the update:

848
- If running Puma/Unicorn, remove the deploy node from the load balancer.
849 850
- If running Sidekiq, ensure the deploy node is not processing jobs:

Evan Read's avatar
Evan Read committed
851
  ```shell
852 853 854 855 856
  sudo gitlab-ctl stop sidekiq
  ```

- If running `geo-logcursor` daemon, ensure the deploy node is not processing events:

Evan Read's avatar
Evan Read committed
857
  ```shell
858 859 860
  sudo gitlab-ctl stop geo-logcursor
  ```

861
For zero-downtime, Puma/Unicorn, Sidekiq, and `geo-logcursor` must be running on other nodes during the update.
862

Evan Read's avatar
Evan Read committed
863
#### Step 2: Update the Geo primary multi-node deployment
864

865
**On all nodes _including_ the primary "deploy node"**
866 867 868 869 870

Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. During software
installation only, this will prevent the upgrade from running
`gitlab-ctl reconfigure` and automatically running database migrations.

Evan Read's avatar
Evan Read committed
871
```shell
872 873 874
sudo touch /etc/gitlab/skip-auto-reconfigure
```

875
**On all other nodes _excluding_ the primary "deploy node"**
876 877 878

1. Ensure that `gitlab_rails['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`.

879 880
1. Ensure nodes are running the latest code

Evan Read's avatar
Evan Read committed
881
   ```shell
882 883 884 885 886 887 888
   sudo gitlab-ctl reconfigure
   ```

**On primary Gitaly only nodes**

1. Update the GitLab package

Evan Read's avatar
Evan Read committed
889
   ```shell
890 891 892 893 894 895 896
   # Debian/Ubuntu
   sudo apt-get update && sudo apt-get install gitlab-ee

   # Centos/RHEL
   sudo yum install gitlab-ee
   ```

897 898
1. Ensure nodes are running the latest code

Evan Read's avatar
Evan Read committed
899
   ```shell
900 901 902
   sudo gitlab-ctl reconfigure
   ```

903
**On the primary "deploy node"**
904 905 906

1. Update the GitLab package

Evan Read's avatar
Evan Read committed
907
   ```shell
908 909 910 911 912 913 914
   # Debian/Ubuntu
   sudo apt-get update && sudo apt-get install gitlab-ee

   # Centos/RHEL
   sudo yum install gitlab-ee
   ```

915
1. To get the regular database migrations and latest code in place, run
916

Evan Read's avatar
Evan Read committed
917
   ```shell
918 919 920
   sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure
   ```

921 922 923 924 925 926
1. If this deploy node is normally used to serve requests or process jobs,
   then you may return it to service at this point.

   - To serve requests, add the deploy node to the load balancer.
   - To process Sidekiq jobs again, start Sidekiq:

Evan Read's avatar
Evan Read committed
927
     ```shell
928 929 930
     sudo gitlab-ctl start sidekiq
     ```

931
**On all other nodes _excluding_ the primary "deploy node"**
932 933 934

1. Update the GitLab package

Evan Read's avatar
Evan Read committed
935
   ```shell
936 937 938 939 940 941 942 943 944
   # Debian/Ubuntu
   sudo apt-get update && sudo apt-get install gitlab-ee

   # Centos/RHEL
   sudo yum install gitlab-ee
   ```

1. Ensure nodes are running the latest code

Evan Read's avatar
Evan Read committed
945
   ```shell
946 947 948
   sudo gitlab-ctl reconfigure
   ```

949
**For all nodes that run Puma/Unicorn or Sidekiq _including_ the primary "deploy node"**
950

951
Hot reload `puma` (or `unicorn`) and `sidekiq` services:
952

Evan Read's avatar
Evan Read committed
953
```shell
954
sudo gitlab-ctl hup puma
955
sudo gitlab-ctl restart sidekiq