docker container gitlab-ce not reachable through the public SSL without any logs
Summary
We use in our organization numbers of Gitlab-CE instance. All of them are updated with latest Gitlab version every 14 days.
We use an external SSL nginx proxy to access our Gitlab.
For all of our Gitlab instancem we encounter issue after a few weeks period of time. Even if we have disk space and ram, the nginx reverse SSL proxy start to timeout and this after a few days. One by one all our gitlab instance end up like this.
We have the following logs in our public reverse ssl nginx proxy
2018/02/19 11:26:51 [error] 14#14: *22353 upstream prematurely closed connection while reading response header from upstream, client: 171.249.81.188, server: module.domain.com, request: "GET /bootstrap-styled/step-progress/ HTTP/2.0", upstream: "http://192.168.0.2:31583/bootstrap-styled/step-progress/", host: "module.domain.com"
2018/02/19 11:26:51 [error] 14#14: *22635 upstream prematurely closed connection while reading response header from upstream, client: 123.143.171.104, server: module.domain.com, request: "GET / HTTP/2.0", upstream: "http://192.168.0.2:31583/", host: "module.domain.com"
2018/02/19 11:26:51 [error] 14#14: *22586 upstream prematurely closed connection while reading response header from upstream, client: 123.143.171.104, server: module.domain.com, request: "HEAD / HTTP/2.0", upstream: "http://192.168.0.2:31583/", host: "module.domain.com"
2018/02/19 11:26:51 [error] 14#14: *22353 upstream prematurely closed connection while reading response header from upstream, client: 171.249.81.188, server: module.domain.com, request: "GET / HTTP/2.0", upstream: "http://192.168.0.2:31583/", host: "module.domain.com"
2018/02/19 11:26:52 [error] 14#14: *22888 connect() failed (111: Connection refused) while connecting to upstream, client: 149.202.200.183, server: module.domain.com, request: "POST /api/v4/jobs/request HTTP/1.1", upstream: "http://192.168.0.2:31583/api/v4/jobs/request", host: "module.domain.com"
2018/02/19 11:26:52 [error] 14#14: *22635 connect() failed (111: Connection refused) while connecting to upstream, client: 123.143.171.104, server: module.domain.com, request: "GET /favicon.ico HTTP/2.0", upstream: "http://192.168.0.2:31583/favicon.ico", host: "module.domain.com", referrer: "https://module.domain.com/"
2018/02/19 11:26:53 [error] 14#14: *22353 connect() failed (111: Connection refused) while connecting to upstream, client: 171.249.81.188, server: module.domain.com, request: "GET /favicon.ico HTTP/2.0", upstream: "http://192.168.0.2:31583/favicon.ico", host: "module.domain.com", referrer: "https://module.domain.com/"
2018/02/19 11:26:53 [error] 14#14: *22891 connect() failed (111: Connection refused) while connecting to upstream, client: 149.202.200.183, server: module.domain.com, request: "POST /api/v4/jobs/request HTTP/1.1", upstream: "http://192.168.0.2:31583/api/v4/jobs/request", host: "module.domain.com"The external nginx ssl reverse proxy looks as follow :
server {
listen 443 ssl http2;
server_name module.domain.com;
server_tokens off;
root /dev/null;
## Increase this if you want to upload large attachments
## Or if you want to accept large git objects over http
client_max_body_size 0;
add_header Strict-Transport-Security max-age=63072000;
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
## include domain ssl config
include /etc/nginx/ssl-conf/wildcard.domain.com.ssl.conf;
## Redirect all request
location / {
gzip off;
## include upstream config
proxy_pass http://192.168.0.2:31583;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_set_header X-Url-Scheme $scheme;
proxy_read_timeout 900;
proxy_send_timeout 900;
}
access_log /var/log/nginx/git_access.log;
error_log /var/log/nginx/git_error.log;
}Gitlab doesn't have relevant logs
This is my gitlab.rb
external_url 'https://module.domain.com'
gitlab_rails['time_zone'] = 'Europe/Paris'
unicorn['worker_timeout'] = 60
unicorn['worker_processes'] = 2
nginx['listen_port'] = 80 # override only if you use a reverse proxy: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#setting-the-nginx-listen-port
nginx['listen_https'] = false # override only if your reverse proxy internally communicates over HTTP: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#supporting-proxied-ssl
nginx['proxy_set_headers'] = {
# "Host" => "$http_host",
"Host" => "$http_host_with_default",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on",
"Upgrade" => "$http_upgrade",
"Connection" => "$connection_upgrade"
}
pages_external_url "https://anotherdomain.com/"
pages_nginx['listen_port'] = 80 # override only if you use a reverse proxy: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#setting-the-nginx-listen-port
pages_nginx['listen_https'] = false # override only if your reverse proxy internally communicates over HTTP: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#supporting-proxied-ssl
pages_nginx['proxy_read_timeout'] = 3600
pages_nginx['proxy_connect_timeout'] = 300
pages_nginx['proxy_set_headers'] = {
"Host" => "$http_host",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on",
"Upgrade" => "$http_upgrade",
"Connection" => "$connection_upgrade"
}
This is the generated internal nginx http proxy configuration
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.
## GitLab
## Modified from https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/gitlab-ssl & https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/gitlab
##
## Lines starting with two hashes (##) are comments with information.
## Lines starting with one hash (#) are configuration parameters that can be uncommented.
##
##################################
## CHUNKED TRANSFER ##
##################################
##
## It is a known issue that Git-over-HTTP requires chunked transfer encoding [0]
## which is not supported by Nginx < 1.3.9 [1]. As a result, pushing a large object
## with Git (i.e. a single large file) can lead to a 411 error. In theory you can get
## around this by tweaking this configuration file and either:
## - installing an old version of Nginx with the chunkin module [2] compiled in, or
## - using a newer version of Nginx.
##
## At the time of writing we do not know if either of these theoretical solutions works.
## As a workaround users can use Git over SSH to push large files.
##
## [0] https://git.kernel.org/cgit/git/git.git/tree/Documentation/technical/http-protocol.txt#n99
## [1] https://github.com/agentzh/chunkin-nginx-module#status
## [2] https://github.com/agentzh/chunkin-nginx-module
##
###################################
## configuration ##
###################################
upstream gitlab-workhorse {
server unix:/var/opt/gitlab/gitlab-workhorse/socket;
}
server {
listen *:80;
server_name module.domain.com;
server_tokens off; ## Don't show the nginx version number, a security best practice
## Increase this if you want to upload large attachments
## Or if you want to accept large git objects over http
client_max_body_size 0;
## Real IP Module Config
## http://nginx.org/en/docs/http/ngx_http_realip_module.html
## HSTS Config
## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
add_header Strict-Transport-Security "max-age=31536000";
## Individual nginx logs for this GitLab vhost
access_log /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
error_log /var/log/gitlab/nginx/gitlab_error.log;
if ($http_host = "") {
set $http_host_with_default "module.domain.com:443";
}
if ($http_host != "") {
set $http_host_with_default $http_host;
}
## If you use HTTPS make sure you disable gzip compression
## to be safe against BREACH attack.
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 3600;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Host $http_host_with_default;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
location ~ (\.git/gitlab-lfs/objects|\.git/info/lfs/objects/batch$) {
proxy_cache off;
proxy_pass http://gitlab-workhorse;
proxy_request_buffering off;
}
location / {
proxy_cache off;
proxy_pass http://gitlab-workhorse;
}
location /assets {
proxy_cache gitlab;
proxy_pass http://gitlab-workhorse;
}
error_page 404 /404.html;
error_page 422 /422.html;
error_page 500 /500.html;
error_page 502 /502.html;
location ~ ^/(404|422|500|502)(-custom)?\.html$ {
root /opt/gitlab/embedded/service/gitlab-rails/public;
internal;
}
}
What is the current bug behavior?
After a few days of service, the backend is not responding through our nginx ssl proxy. Without reconfiguring nginx, just by restarting gitlab, we get our service back.
What is the expected correct behavior?
Gitlab should always be reachable.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
sudo gitlab-rake gitlab:env:info:
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.6p384
Gem Version: 2.6.13
Bundler Version:1.13.7
Rake Version: 12.3.0
Redis Version: 3.2.11
Git Version: 2.14.3
Sidekiq Version:5.0.5
Go Version: unknown
GitLab information
Version: 10.4.3
Revision: 183dd5d
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://module.domain.com
HTTP Clone URL: https://module.domain.com/some-group/some-project.git
SSH Clone URL: ssh://git@module.domain.com:20024/some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 5.11.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
gitlab-rake gitlab:check SANITIZE=true :
Checking GitLab Shell ...
GitLab Shell version >= 5.11.0 ? ... OK (5.11.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
5/1 ... ok
5/2 ... ok
6/3 ... ok
7/5 ... ok
6/6 ... ok
2/7 ... repository is empty
9/12 ... ok
9/14 ... ok
9/15 ... ok
9/16 ... ok
6/52 ... ok
9/53 ... ok
6/54 ... ok
9/56 ... ok
12/59 ... ok
6/62 ... ok
6/63 ... ok
7/64 ... ok
9/66 ... ok
13/68 ... ok
6/69 ... ok
12/70 ... ok
14/71 ... ok
12/72 ... ok
6/73 ... ok
6/74 ... ok
6/75 ... ok
17/76 ... ok
13/77 ... ok
18/78 ... ok
4/80 ... ok
19/81 ... ok
13/84 ... ok
20/86 ... ok
20/87 ... ok
20/88 ... ok
13/90 ... ok
17/91 ... ok
13/92 ... ok
22/93 ... ok
3/94 ... ok
13/95 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
5/1 ... yes
5/2 ... yes
6/3 ... yes
7/5 ... yes
6/6 ... yes
2/7 ... yes
9/12 ... yes
9/14 ... yes
9/15 ... yes
9/16 ... yes
6/52 ... yes
9/53 ... yes
6/54 ... yes
9/56 ... yes
12/59 ... yes
6/62 ... yes
6/63 ... yes
7/64 ... yes
9/66 ... yes
13/68 ... yes
6/69 ... yes
12/70 ... yes
14/71 ... yes
12/72 ... yes
6/73 ... yes
6/74 ... yes
6/75 ... yes
17/76 ... yes
13/77 ... yes
18/78 ... yes
4/80 ... yes
19/81 ... yes
13/84 ... yes
20/86 ... yes
20/87 ... yes
20/88 ... yes
13/90 ... yes
17/91 ... yes
13/92 ... yes
22/93 ... yes
3/94 ... yes
13/95 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.6)
Git version >= 2.7.3 ? ... yes (2.14.3)
Git user has default SSH configuration? ... yes
Active users: ... 5
Possible fixes
We have no clue, gitlab is not giving any log and because it is configured through https, we can't really access through the internal http the gitlab because of the wrong protocol, but the server is answering.
Edited by Kopax Anderson