nginx.md 29 KB
Newer Older
1 2 3 4 5 6
---
stage: Enablement
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---

Jacob Vosmaer's avatar
Jacob Vosmaer committed
7 8
# NGINX settings

9 10 11
## Service-specific NGINX settings

Users can configure NGINX settings differently for different services via
12
`gitlab.rb`. Settings for the GitLab Rails application can be configured using the
13 14 15 16 17
`nginx['<some setting>']` keys. There are similar keys for other services like
`pages_nginx`, `mattermost_nginx` and `registry_nginx`. All the configurations
available for `nginx` are also available for these `<service_nginx>` settings and
share the same default values as GitLab NGINX.

18
If modifying via `gitlab.rb`, users have to configure NGINX setting for each
19
service separately. Settings given via `nginx['foo']` WILL NOT be replicated to
20
service specific NGINX configuration (as `registry_nginx['foo']` or
21 22 23 24 25 26 27 28 29 30
`mattermost_nginx['foo']`, etc.). For example, to configure HTTP to HTTPS
redirection for GitLab, Mattermost and Registry, the following settings should
be added to `gitlab.rb`

```ruby
nginx['redirect_http_to_https'] = true
registry_nginx['redirect_http_to_https'] = true
mattermost_nginx['redirect_http_to_https'] = true
```

Ben Bodenmiller's avatar
Ben Bodenmiller committed
31
NOTE: **Note:** Modifying NGINX configuration should be done with care as incorrect
32 33
or incompatible configuration may yield to unavailability of service.

34 35
## Enable HTTPS

36 37
By default, Omnibus GitLab does not use HTTPS. If you want to enable HTTPS for
`gitlab.example.com`, there are two options:
38

39
1. [Free and automated HTTPS with Let's Encrypt](ssl.md#lets-encrypt-integration)
40 41
1. [Manually configuring HTTPS with your own certificates](#manually-configuring-https)

42 43
### Warning

Evan Read's avatar
Evan Read committed
44
The NGINX configuration will tell browsers and clients to only communicate with your
45 46
GitLab instance over a secure connection for the next 24 months. By enabling
HTTPS you'll need to provide a secure connection to your instance for at least
47 48
the next 24 months.

49 50
## Manually configuring HTTPS

51
By default, Omnibus GitLab does not use HTTPS.
52

53 54 55 56 57 58 59 60 61 62
To enable HTTPS for the domain `gitlab.example.com`:

1. Edit the `external_url` in `/etc/gitlab/gitlab.rb`:

   ```ruby
   # note the 'https' below
   external_url "https://gitlab.example.com"
   ```

1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
63

Evan Read's avatar
Evan Read committed
64
   ```shell
65
   sudo mkdir -p /etc/gitlab/ssl
66
   sudo chmod 755 /etc/gitlab/ssl
67 68 69
   sudo cp gitlab.example.com.key gitlab.example.com.crt /etc/gitlab/ssl/
   ```

70
   Because the hostname in our example is `gitlab.example.com`, Omnibus GitLab
71
   will look for private key and public certificate files called
72 73
   `/etc/gitlab/ssl/gitlab.example.com.key` and `/etc/gitlab/ssl/gitlab.example.com.crt`,
   respectively.
74

75 76 77 78
   Make sure you use the full certificate chain in order to prevent SSL errors when
   clients connect. The full certificate chain order should consist of the server certificate first,
   followed by all intermediate certificates, with the root CA last.

79
   If the `certificate.key` file is password protected, NGINX will not ask for
80 81 82
   the password when you reconfigure GitLab. In that case, Omnibus GitLab will
   fail silently with no error messages. To remove the password from the key, run:

Evan Read's avatar
Evan Read committed
83
   ```shell
84 85 86 87
   openssl rsa -in certificate_before.key -out certificate_after.key
   ```

1. Now, reconfigure GitLab:
88

Evan Read's avatar
Evan Read committed
89
   ```shell
90 91
   sudo gitlab-ctl reconfigure
   ```
92

93
When the reconfigure finishes, your GitLab instance should be reachable at `https://gitlab.example.com`.
94

95 96 97
If you are using a firewall you may have to open port 443 to allow inbound
HTTPS traffic.

98
```shell
99 100 101 102 103 104
# UFW example (Debian, Ubuntu)
sudo ufw allow https

# lokkit example (RedHat, CentOS 6)
sudo lokkit -s https

105
# firewall-cmd (RedHat, Centos 7)
106 107 108 109
sudo firewall-cmd --permanent --add-service=https
sudo systemctl reload firewalld
```

Jacob Vosmaer's avatar
Jacob Vosmaer committed
110 111
## Redirect `HTTP` requests to `HTTPS`

Evan Read's avatar
Evan Read committed
112
By default, when you specify an `external_url` starting with 'https', NGINX will
Jacob Vosmaer's avatar
Jacob Vosmaer committed
113 114 115 116
no longer listen for unencrypted HTTP traffic on port 80. If you want to
redirect all HTTP traffic to HTTPS you can use the `redirect_http_to_https`
setting.

117 118
NOTE: **Note:** This behavior is enabled by default.

Jacob Vosmaer's avatar
Jacob Vosmaer committed
119 120 121 122 123 124 125 126
```ruby
external_url "https://gitlab.example.com"
nginx['redirect_http_to_https'] = true
```

## Change the default port and the SSL certificate locations

If you need to use an HTTPS port other than the default (443), just specify it
Evan Read's avatar
Evan Read committed
127
as part of the `external_url`.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
128 129 130 131 132

```ruby
external_url "https://gitlab.example.com:2443"
```

133 134 135
To set the location of ssl certificates create `/etc/gitlab/ssl` directory,
place the `.crt` and `.key` files in the directory and specify the following
configuration:
136 137 138

```ruby
# For GitLab
René Genz's avatar
René Genz committed
139
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.example.com.crt"
140 141 142
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.example.com.key"
```

Jacob Vosmaer's avatar
Jacob Vosmaer committed
143 144
Run `sudo gitlab-ctl reconfigure` for the change to take effect.

145 146
## Update the SSL Certificates

Evan Read's avatar
Evan Read committed
147
If the content of your SSL certificates has been updated, but no configuration
148
changes have been made to `gitlab.rb`, then `gitlab-ctl reconfigure` will not
Evan Read's avatar
Evan Read committed
149 150
affect NGINX. Instead, run `sudo gitlab-ctl hup nginx` to cause NGINX to
[reload the existing configuration and new certificates](http://nginx.org/en/docs/control.html)
151 152
gracefully.

153 154
## Change the default proxy headers

155
By default, when you specify `external_url` Omnibus GitLab will set a few
156 157
NGINX proxy headers that are assumed to be sane in most environments.

158
For example, Omnibus GitLab will set:
159

160
```plaintext
161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
```

if you have specified `https` schema in the `external_url`.

However, if you have a situation where your GitLab is in a more complex setup
like behind a reverse proxy, you will need to tweak the proxy headers in order
to avoid errors like `The change you wanted was rejected` or
`Can't verify CSRF token authenticity Completed 422 Unprocessable`.

This can be achieved by overriding the default headers, eg. specify
in `/etc/gitlab/gitlab.rb`:

```ruby
 nginx['proxy_set_headers'] = {
  "X-Forwarded-Proto" => "http",
  "CUSTOM_HEADER" => "VALUE"
 }
```

182
Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure)
183 184 185 186
for the changes to take effect.

This way you can specify any header supported by NGINX you require.

187
## Configuring GitLab `trusted_proxies` and the NGINX `real_ip` module
188

189
By default, NGINX and GitLab will log the IP address of the connected client.
190 191 192 193 194 195 196 197 198 199

If your GitLab is behind a reverse proxy, you may not want the IP address of
the proxy to show up as the client address.

You can have NGINX look for a different address to use by adding your reverse
proxy to the `real_ip_trusted_addresses` list:

```ruby
# Each address is added to the the NGINX config as 'set_real_ip_from <address>;'
nginx['real_ip_trusted_addresses'] = [ '192.168.1.0/24', '192.168.2.1', '2001:0db8::/32' ]
200
# other real_ip config options
201
nginx['real_ip_header'] = 'X-Forwarded-For'
202
nginx['real_ip_recursive'] = 'on'
203 204 205
```

Description of the options:
Evan Read's avatar
Evan Read committed
206 207

- <http://nginx.org/en/docs/http/ngx_http_realip_module.html>
208

209
By default, Omnibus GitLab will use the IP addresses in `real_ip_trusted_addresses`
210 211 212
as GitLab's trusted proxies, which will keep users from being listed as signed
in from those IPs.

213
Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure)
214 215
for the changes to take effect.

216 217
## Configuring HTTP2 protocol

218
By default, when you specify that your GitLab instance should be reachable
219
through HTTPS by specifying `external_url "https://gitlab.example.com"`,
Evan Read's avatar
Evan Read committed
220
[http2 protocol](https://tools.ietf.org/html/rfc7540) is also enabled.
221

222
The Omnibus GitLab package sets required ssl_ciphers that are compatible with
223 224 225
http2 protocol.

If you are specifying custom ssl_ciphers in your configuration and a cipher is
Evan Read's avatar
Evan Read committed
226
in [http2 cipher blacklist](https://tools.ietf.org/html/rfc7540#appendix-A), once you try to reach your GitLab instance you will
227 228 229 230 231
be presented with `INADEQUATE_SECURITY` error in your browser.

Consider removing the offending ciphers from the cipher list. Changing ciphers
is only necessary if you have a very specific custom setup.

Evan Read's avatar
Evan Read committed
232
For more information on why you would want to have http2 protocol enabled, check out
Evan Read's avatar
Evan Read committed
233
the [http2 whitepaper](https://assets.wp.nginx.com/wp-content/uploads/2015/09/NGINX_HTTP2_White_Paper_v4.pdf?_ga=1.127086286.212780517.1454411744).
234 235 236 237 238 239 240 241

If changing the ciphers is not an option you can disable http2 support by
specifying in `/etc/gitlab/gitlab.rb`:

```ruby
nginx['http2_enabled'] = false
```

242
Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure)
243 244
for the changes to take effect.

245 246
NOTE: **Note:** The `http2` setting only works for the main GitLab application and not for the other services.

Jacob Vosmaer's avatar
Jacob Vosmaer committed
247 248
## Using a non-bundled web-server

249 250
By default, Omnibus GitLab installs GitLab with bundled NGINX.
Omnibus GitLab allows webserver access through the `gitlab-www` user, which resides
Jacob Vosmaer's avatar
Jacob Vosmaer committed
251
in the group with the same name. To allow an external webserver access to
252
GitLab, the external webserver user needs to be added to the `gitlab-www` group.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
253

254
To use another web server like Apache or an existing NGINX installation you
255
will have to perform the following steps:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
256

257
1. **Disable bundled NGINX**
Jacob Vosmaer's avatar
Jacob Vosmaer committed
258

259
   In `/etc/gitlab/gitlab.rb` set:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
260

261 262 263
   ```ruby
   nginx['enable'] = false
   ```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
264

265
1. **Set the username of the non-bundled web-server user**
Jacob Vosmaer's avatar
Jacob Vosmaer committed
266

267
   By default, Omnibus GitLab has no default setting for the external webserver
268
   user, you have to specify it in the configuration. For Debian/Ubuntu the
269 270
   default user is `www-data` for both Apache/NGINX whereas for RHEL/CentOS
   the NGINX user is `nginx`.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
271

272
   *Note: Make sure you have first installed Apache/NGINX so the webserver user is created, otherwise omnibus will fail while reconfiguring.*
Jacob Vosmaer's avatar
Jacob Vosmaer committed
273

274 275
   Let's say for example that the webserver user is `www-data`.
   In `/etc/gitlab/gitlab.rb` set:
276

277 278 279
   ```ruby
   web_server['external_users'] = ['www-data']
   ```
280

281
   *Note: This setting is an array so you can specify more than one user to be added to `gitlab-www` group.*
282

283
   Run `sudo gitlab-ctl reconfigure` for the change to take effect.
284

Evan Read's avatar
Evan Read committed
285
   *Note: if you are using SELinux and your web server runs under a restricted SELinux profile you may have to [loosen the restrictions on your web server](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache#selinux-modifications).*
286

287
   *Note: make sure that the webserver user has the correct permissions on all directories used by external web-server, otherwise you will receive `failed (XX: Permission denied) while reading upstream` errors.
288

289 290
1. **Add the non-bundled web-server to the list of trusted proxies**

291
   Normally, Omnibus GitLab defaults the list of trusted proxies to what was
292
   configured in the `real_ip` module for the bundled NGINX.
293

294 295 296
   For non-bundled web-servers the list needs to be configured directly, and should
   include the IP address of your web-server if it is not on the same machine as GitLab.
   Otherwise, users will be shown as being signed in from your web-server's IP address.
297

298 299 300
   ```ruby
   gitlab_rails['trusted_proxies'] = [ '192.168.1.0/24', '192.168.2.1', '2001:0db8::/32' ]
   ```
301

302
1. **(Optional) Set the right GitLab Workhorse settings if using Apache**
303

304
   *Note: The values below were added in GitLab 8.2, make sure you have the latest version installed.*
305

306
   Apache cannot connect to a UNIX socket but instead needs to connect to a
307
   TCP Port. To allow GitLab Workhorse to listen on TCP (by default port 8181)
308
   edit `/etc/gitlab/gitlab.rb`:
309

310
   ```ruby
311 312 313
   gitlab_workhorse['listen_network'] = "tcp"
   gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"
   ```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
314

315
   Run `sudo gitlab-ctl reconfigure` for the change to take effect.
316 317 318

1. **Download the right web server configs**

Evan Read's avatar
Evan Read committed
319
   Go to [GitLab recipes repository](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server) and look for the omnibus
320 321 322 323 324
   configs in the webserver directory of your choice. Make sure you pick the
   right configuration file depending whether you choose to serve GitLab with
   SSL or not. The only thing you need to change is `YOUR_SERVER_FQDN` with
   your own FQDN and if you use SSL, the location where your SSL keys currently
   reside. You also might need to change the location of your log files.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
325

326 327 328 329 330 331
## Setting the NGINX listen address or addresses

By default NGINX will accept incoming connections on all local IPv4 addresses.
You can change the list of addresses in `/etc/gitlab/gitlab.rb`.

```ruby
332 333 334 335 336
 # Listen on all IPv4 and IPv6 addresses
nginx['listen_addresses'] = ["0.0.0.0", "[::]"]
registry_nginx['listen_addresses'] = ['*', '[::]']
mattermost_nginx['listen_addresses'] = ['*', '[::]']
pages_nginx['listen_addresses'] = ['*', '[::]']
337 338
```

339 340 341 342 343
## Setting the NGINX listen port

By default NGINX will listen on the port specified in `external_url` or
implicitly use the right port (80 for HTTP, 443 for HTTPS). If you are running
GitLab behind a reverse proxy, you may want to override the listen port to
Evan Read's avatar
Evan Read committed
344
something else. For example, to use port 8081:
345 346

```ruby
347
nginx['listen_port'] = 8081
348 349
```

350 351 352
## Supporting proxied SSL

By default NGINX will auto-detect whether to use SSL if `external_url`
Evan Read's avatar
Evan Read committed
353
contains `https://`. If you are running GitLab behind a reverse proxy, you
354 355 356
may wish to terminate SSL at another proxy server or load balancer. To do this,
be sure the `external_url` contains `https://` and apply the following
configuration to `gitlab.rb`:
357 358

```ruby
359
nginx['listen_port'] = 80
360 361 362
nginx['listen_https'] = false
```

363 364 365 366 367 368 369 370 371 372 373 374 375 376
Other bundled components (Registry, Pages, etc) use a similar strategy for
proxied SSL. Set the particular component's `*_external_url` with `https://` and
prefix the `nginx[...]` configuration with the component name. For example, for
Registry use the following configuration:

```ruby
registry_external_url 'https://registry.example.com'

registry_nginx['listen_port'] = 80
registry_nginx['listen_https'] = false
```

The same format can be used for Pages (`pages_` prefix) and Mattermost (`mattermost_` prefix).

377 378
Note that you may need to configure your reverse proxy or load balancer to
forward certain headers (e.g. `Host`, `X-Forwarded-Ssl`, `X-Forwarded-For`,
379
`X-Forwarded-Port`) to GitLab (and Mattermost if you use one). You may see improper redirections or errors
380 381
(e.g. "422 Unprocessable Entity", "Can't verify CSRF token authenticity") if
you forget this step. For more information, see:
382

Marcel Amirault's avatar
Marcel Amirault committed
383
- <https://stackoverflow.com/questions/16042647/whats-the-de-facto-standard-for-a-reverse-proxy-to-tell-the-backend-ssl-is-used>
384
- <https://websiteforstudents.com/setup-apache2-reverse-proxy-nginx-ubuntu-17-04-17-10/>
385

386 387 388
## Setting HTTP Strict Transport Security

By default GitLab enables Strict Transport Security which informs browsers that
389 390 391
they should only contact the website using HTTPS. When a browser visits a
GitLab instance even once, it will remember to no longer attempt insecure connections,
even when the user is explicitly entering a `http://` URL. Such a URL will be automatically redirected by the browser to `https://` variant.
392 393

```ruby
394 395
nginx['hsts_max_age'] = 31536000
nginx['hsts_include_subdomains'] = false
396 397
```

398
By default `max_age` is set for one year, this is how long browser will remember to only connect through HTTPS.
399 400
Setting `max_age` to 0 will disable this feature. For more information see:

Evan Read's avatar
Evan Read committed
401
- <https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/>
402

403 404
NOTE: **Note:** The HSTS settings only work for the main GitLab application and not for the other services.

405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425
## Setting the Referrer-Policy header

By default, GitLab sets the `Referrer-Policy` header to `strict-origin-when-cross-origin` on all responses.

This makes the client send the full URL as referrer when making a same-origin request but only send the
origin when making cross-origin requests.

To set this header to a different value:

```ruby
nginx['referrer_policy'] = 'same-origin'
```

You can also disable this header to make the client use its default setting:

```ruby
nginx['referrer_policy'] = false
```

Note that setting this to `origin` or `no-referrer` would break some features in GitLab that require the full referrer URL.

Evan Read's avatar
Evan Read committed
426
- <https://www.w3.org/TR/referrer-policy/>
427

Evan Read's avatar
Evan Read committed
428
## Disabling Gzip compression
Stan Hu's avatar
Stan Hu committed
429

Evan Read's avatar
Evan Read committed
430
By default, GitLab enables Gzip compression for text data over 10240 bytes. To
Stan Hu's avatar
Stan Hu committed
431 432 433 434 435 436
disable this behavior:

```ruby
nginx['gzip_enabled'] = false
```

437 438
NOTE: **Note:** The `gzip` setting only works for the main GitLab application and not for the other services.

439
## Using custom SSL ciphers
440

441
By default GitLab is using SSL ciphers that are combination of testing on <https://gitlab.com> and various best practices contributed by the GitLab community.
442 443 444 445 446 447 448

However, you can change the ssl ciphers by adding to `gitlab.rb`:

```ruby
  nginx['ssl_ciphers'] = "CIPHER:CIPHER1"
```

449
and running reconfigure.
450 451 452 453 454 455

You can also enable `ssl_dhparam` directive.

First, generate `dhparams.pem` with `openssl dhparam -out dhparams.pem 2048`. Then, in `gitlab.rb` add a path to the generated file, for example:

```ruby
456
  nginx['ssl_dhparam'] = "/etc/gitlab/ssl/dhparams.pem"
457 458
```

459
After the change run `sudo gitlab-ctl reconfigure`.
460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478

## Enable 2-way SSL Client Authentication

To require web clients to authenticate with a trusted certificate, you can enable 2-way SSL by adding to `gitlab.rb`:

```ruby
  nginx['ssl_verify_client'] = "on"
```

and running reconfigure.

These additional options NGINX supports for configuring SSL client authentication can also be configured:

```ruby
  nginx['ssl_client_certificate'] = "/etc/pki/tls/certs/root-certs.pem"
  nginx['ssl_verify_depth'] = "2"
```

After making the changes run `sudo gitlab-ctl reconfigure`.
479

Jacob Vosmaer's avatar
Jacob Vosmaer committed
480 481
## Inserting custom NGINX settings into the GitLab server block

482 483 484
Please keep in mind that these custom settings may create conflicts if the
same settings are defined in your `gitlab.rb` file.

Jacob Vosmaer's avatar
Jacob Vosmaer committed
485 486 487 488 489 490 491 492 493 494
If you need to add custom settings into the NGINX `server` block for GitLab for
some reason you can use the following setting.

```ruby
# Example: block raw file downloads from a specific repository
nginx['custom_gitlab_server_config'] = "location ^~ /foo-namespace/bar-project/raw/ {\n deny all;\n}\n"
```

Run `gitlab-ctl reconfigure` to rewrite the NGINX configuration and restart
NGINX.
495

496 497 498
This inserts the defined string into the end of the `server` block of
`/var/opt/gitlab/nginx/conf/gitlab-http.conf`.

Marcel Amirault's avatar
Marcel Amirault committed
499
### Notes
500

501
- If you're adding a new location, you might need to include
502

503 504 505 506
  ```conf
  proxy_cache off;
  proxy_pass http://gitlab-workhorse;
  ```
507

Evan Read's avatar
Evan Read committed
508
  in the string or in the included NGINX configuration. Without these, any sub-location
509
  will return a 404. See
Marcel Amirault's avatar
Marcel Amirault committed
510
  [GitLab CE Issue #30619](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30619).
511
- You cannot add the root `/` location or the `/assets` location as those already
512
  exist in `gitlab-http.conf`.
513

Evan Read's avatar
Evan Read committed
514
## Inserting custom settings into the NGINX configuration
515

Evan Read's avatar
Evan Read committed
516
If you need to add custom settings into the NGINX configuration, for example to include
517 518 519 520 521 522 523 524 525 526
existing server blocks, you can use the following setting.

```ruby
# Example: include a directory to scan for additional config files
nginx['custom_nginx_config'] = "include /etc/nginx/conf.d/*.conf;"
```

Run `gitlab-ctl reconfigure` to rewrite the NGINX configuration and restart
NGINX.

527 528 529
This inserts the defined string into the end of the `http` block of
`/var/opt/gitlab/nginx/conf/nginx.conf`.

530
## Custom error pages
531

532 533 534 535
You can use `custom_error_pages` to modify text on the default GitLab error page.
This can be used for any valid HTTP error code; e.g 404, 502.

As an example the following would modify the default 404 error page.
536 537

```ruby
538
nginx['custom_error_pages'] = {
539 540 541 542 543 544 545 546
  '404' => {
    'title' => 'Example title',
    'header' => 'Example header',
    'message' => 'Example message'
  }
}
```

547 548 549 550
This would result in the 404 error page below.

![custom 404 error page](img/error_page_example.png)

551 552 553
Run `gitlab-ctl reconfigure` to rewrite the NGINX configuration and restart
NGINX.

554
## Using an existing Passenger/NGINX installation
555

556
In some cases you may want to host GitLab using an existing Passenger/NGINX
557 558 559
installation but still have the convenience of updating and installing using
the omnibus packages.

560 561 562 563
NOTE: **Note:** When disabling NGINX, you won't be able to access
other services included by Omnibus, like Grafana, Mattermost, etc. unless
you manually add them in `nginx.conf`.

564 565
### Configuration

566
First, you'll need to setup your `/etc/gitlab/gitlab.rb` to disable the built-in
567
NGINX and Puma:
568 569

```ruby
570 571 572
# Define the external url
external_url 'http://git.example.com'

573 574 575
# Disable the built-in nginx
nginx['enable'] = false

576 577
# Disable the built-in puma
puma['enable'] = false
578 579

# Set the internal API URL
580 581 582 583
gitlab_rails['internal_api_url'] = 'http://git.example.com'

# Define the web server process user (ubuntu/nginx)
web_server['external_users'] = ['www-data']
584 585 586 587
```

Make sure you run `sudo gitlab-ctl reconfigure` for the changes to take effect.

588
**Note:** If you are running a version older than 8.16.0, you will have to
589
manually remove the Unicorn service file (`/opt/gitlab/service/unicorn`), if
590 591
exists, for reconfigure to succeed.

592 593
### Vhost (server block)

594
Then, in your custom Passenger/NGINX installation, create the following site
595
configuration file:
596

597
```plaintext
598 599
upstream gitlab-workhorse {
  server unix://var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0;
600 601
}

602 603
server {
  listen *:80;
604
  server_name git.example.com;
605 606 607 608 609 610 611 612 613 614 615 616
  server_tokens off;
  root /opt/gitlab/embedded/service/gitlab-rails/public;

  client_max_body_size 250m;

  access_log  /var/log/gitlab/nginx/gitlab_access.log;
  error_log   /var/log/gitlab/nginx/gitlab_error.log;

  # Ensure Passenger uses the bundled Ruby version
  passenger_ruby /opt/gitlab/embedded/bin/ruby;

  # Correct the $PATH variable to included packaged executables
617
  passenger_env_var PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin";
618 619 620 621 622 623 624 625 626 627

  # Make sure Passenger runs as the correct user and group to
  # prevent permission issues
  passenger_user git;
  passenger_group git;

  # Enable Passenger & keep at least one instance running at all times
  passenger_enabled on;
  passenger_min_instances 1;

628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660
  location ~ ^/[\w\.-]+/[\w\.-]+/(info/refs|git-upload-pack|git-receive-pack)$ {
    # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
    error_page 418 = @gitlab-workhorse;
    return 418;
  }

  location ~ ^/[\w\.-]+/[\w\.-]+/repository/archive {
    # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
    error_page 418 = @gitlab-workhorse;
    return 418;
  }

  location ~ ^/api/v3/projects/.*/repository/archive {
    # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
    error_page 418 = @gitlab-workhorse;
    return 418;
  }

  # Build artifacts should be submitted to this location
  location ~ ^/[\w\.-]+/[\w\.-]+/builds/download {
      client_max_body_size 0;
      # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
      error_page 418 = @gitlab-workhorse;
      return 418;
  }

  # Build artifacts should be submitted to this location
  location ~ /ci/api/v1/builds/[0-9]+/artifacts {
      client_max_body_size 0;
      # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
      error_page 418 = @gitlab-workhorse;
      return 418;
  }
661

662 663 664 665 666 667 668 669
  # Build artifacts should be submitted to this location
  location ~ /api/v4/jobs/[0-9]+/artifacts {
      client_max_body_size 0;
      # 'Error' 418 is a hack to re-use the @gitlab-workhorse block
      error_page 418 = @gitlab-workhorse;
      return 418;
  }

670

671
  # For protocol upgrades from HTTP/1.0 to HTTP/1.1 we need to provide Host header if its missing
672
  if ($http_host = "") {
673 674
  # use one of values defined in server_name
    set $http_host_with_default "git.example.com";
675 676 677 678 679 680
  }

  if ($http_host != "") {
    set $http_host_with_default $http_host;
  }

681
  location @gitlab-workhorse {
682 683 684

    ## https://github.com/gitlabhq/gitlabhq/issues/694
    ## Some requests take more than 30 seconds.
685
    proxy_read_timeout      3600;
686 687 688 689 690 691
    proxy_connect_timeout   300;
    proxy_redirect          off;

    # Do not buffer Git HTTP responses
    proxy_buffering off;

692
    proxy_set_header    Host                $http_host_with_default;
693 694 695
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;
696

697
    proxy_pass http://gitlab-workhorse;
698

699 700 701 702 703
    ## The following settings only work with NGINX 1.7.11 or newer
    #
    ## Pass chunked request bodies to gitlab-workhorse as-is
    # proxy_request_buffering off;
    # proxy_http_version 1.1;
704 705
  }

706 707 708 709 710 711 712 713 714 715
  ## Enable gzip compression as per rails guide:
  ## http://guides.rubyonrails.org/asset_pipeline.html#gzip-compression
  ## WARNING: If you are using relative urls remove the block below
  ## See config/application.rb under "Relative url support" for the list of
  ## other files that need to be changed for relative url support
  location ~ ^/(assets)/ {
    root /opt/gitlab/embedded/service/gitlab-rails/public;
    gzip_static on; # to serve pre-gzipped version
    expires max;
    add_header Cache-Control public;
716 717
  }

718 719 720 721 722
  ## To access Grafana
  location /-/grafana/ {
    proxy_pass http://localhost:3000/;
  }

723
  error_page 502 /502.html;
724 725
}
```
726

727
Don't forget to update `git.example.com` in the above example to be your server URL.
728

729
**Note:** If you wind up with a 403 forbidden, it's possible that you haven't enabled passenger in `/etc/nginx/nginx.conf`, to do so simply uncomment:
730

731
```plaintext
732 733
# include /etc/nginx/passenger.conf;
```
734

735
then, `sudo service nginx reload`
736

Ben Bodenmiller's avatar
Ben Bodenmiller committed
737
## Enabling/Disabling nginx_status
738

739
By default you will have an NGINX health-check endpoint configured at `127.0.0.1:8060/nginx_status` to monitor your NGINX server status.
740

Marcel Amirault's avatar
Marcel Amirault committed
741
### The following information will be displayed
742

743
```plaintext
744 745 746 747 748
Active connections: 1
server accepts handled requests
 18 18 36
Reading: 0 Writing: 1 Waiting: 0
```
Evan Read's avatar
Evan Read committed
749 750 751 752 753 754

- Active connections – Open connections in total.
- 3 figures are shown.
  - All accepted connections.
  - All handled connections.
  - Total number of handled requests.
755 756
- Reading: NGINX reads request headers
- Writing: NGINX reads request bodies, processes requests, or writes responses to a client
Evan Read's avatar
Evan Read committed
757
- Waiting: Keep-alive connections. This number depends on the keepalive-timeout.
758

Ben Bodenmiller's avatar
Ben Bodenmiller committed
759
### Configuration options
760

761
Edit `/etc/gitlab/gitlab.rb`:
762

763
```ruby
764 765 766 767 768 769
nginx['status'] = {
  "listen_addresses" => ["127.0.0.1"],
  "fqdn" => "dev.example.com",
  "port" => 9999,
  "options" => {
    "access_log" => "on", # Disable logs for stats
770
    "allow" => "127.0.0.1", # Only allow access from localhost
771 772
    "deny" => "all" # Deny access to anyone else
  }
773 774 775 776 777 778
}
```

If you don't find this service useful for your current infrastructure you can disable it with:

```ruby
779 780 781
nginx['status'] = {
  'enable' => false
}
782 783
```

784
Make sure you run `sudo gitlab-ctl reconfigure` for the changes to take effect.
785

786 787
#### Warning

788
To ensure that user uploads are accessible your NGINX user (usually `www-data`)
789 790 791 792 793 794
should be added to the `gitlab-www` group. This can be done using the following command:

```shell
sudo usermod -aG gitlab-www www-data
```

Ben Bodenmiller's avatar
Ben Bodenmiller committed
795
## Templates
796

797
Other than the Passenger configuration in place of Unicorn and the lack of HTTPS
Evan Read's avatar
Evan Read committed
798
(although this could be enabled) these files are mostly identical to:
799

Evan Read's avatar
Evan Read committed
800
- [Bundled GitLab NGINX configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb)
801

802
Don't forget to restart NGINX to load the new configuration (on Debian-based
803 804
systems `sudo service nginx restart`).

Ben Bodenmiller's avatar
Ben Bodenmiller committed
805
## Troubleshooting
806

Ben Bodenmiller's avatar
Ben Bodenmiller committed
807
### 400 Bad Request: too many Host headers
Evan Read's avatar
Evan Read committed
808

809 810
Make sure you don't have the `proxy_set_header` configuration in
`nginx['custom_gitlab_server_config']` settings and instead use the
Evan Read's avatar
Evan Read committed
811
['proxy_set_headers'](#supporting-proxied-ssl) configuration in your `gitlab.rb` file.
812

Evan Read's avatar
Evan Read committed
813
### `javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure`
814

815
Starting with GitLab 10, the Omnibus GitLab package no longer supports TLSv1 protocol by default.
816 817 818
This can cause connection issues with some older Java based IDE clients when interacting with
your GitLab instance.
We strongly urge you to upgrade ciphers on your server, similar to what was mentioned
Marcel Amirault's avatar
Marcel Amirault committed
819
in [this user comment](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/624#note_299061).
820 821

If it is not possible to make this server change, you can default back to the old
Yoginth's avatar
Yoginth committed
822
behavior by changing the values in your `/etc/gitlab/gitlab.rb`:
823

824
```ruby
825
nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
826 827
```

828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847
### Mismatch between private key and certificate

If you see `x509 certificate routines:X509_check_private_key:key values mismatch)` in the NGINX logs (`/var/log/gitlab/nginx/current` by default for Omnibus), there is a mismatch between your private key and certificate.

To fix this, you will need to match the correct private key with your certificate.

To ensure you have the correct key and certificate, you can ensure that the modulus of the private key and certificate match:

```shell
/opt/gitlab/embedded/bin/openssl rsa -in /etc/gitlab/ssl/gitlab.example.com.key -noout -modulus | openssl sha1

/opt/gitlab/embedded/bin/openssl x509 -in /etc/gitlab/ssl/gitlab.example.com.crt -noout -modulus| openssl sha1
```

Once you verify that they match, you will need to reconfigure and reload NGINX:

```shell
sudo gitlab-ctl reconfigure
sudo gitlab-ctl hup nginx
```