GitLab Custom Domains not working when pages is on seperate server
Summary
We are using a self hosted gitlab instance with gitlab pages on a separate server, as described here:
https://docs.gitlab.com/ee/administration/pages/index.html#running-gitlab-pages-on-a-separate-server. We want to use the custom domain feature and also the wildcard domain feature.
Wildcard domains for pages are working. But we cannot enable custom domains on our project settings.
Steps to reproduce
- Configure pages as described in documentation https://docs.gitlab.com/ee/administration/pages/index.html#running-gitlab-pages-on-a-separate-server.
- Open a project with deployed pages as user with owner permission level
- We get the message support for domains & certificates is disabled as pictured below:
What is the current bug behavior?
- Cannot enable gitlab custom domain feature
What is the expected correct behavior?
- Beeing able to enable and use custom domain feature
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
On gitlab server
:~$ sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 20.04
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.3
Redis Version: 6.0.12
Git Version: 2.31.1
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.11.2
Revision: d8d57a90208
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.6
URL: https://gitlab.domain.tld
HTTP Clone URL: https://gitlab.domain.tld/some-group/some-project.git
SSH Clone URL: git@gitlab.domain.tld:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: azure_activedirectory_v2
GitLab Shell
Version: 13.17.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
On pages server:
rz20adm@gitlab-pages-0001:~$ sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 20.04
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.3
Redis Version: 6.0.12
Git Version: 2.31.1
Sidekiq Version:5.2.9
Go Version: unknown
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)
Results of GitLab application Check
gitlab server
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.31.1)
Git user has default SSH configuration? ... yes
Active users: ... 106
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
gitlab pages
rz20adm@gitlab-pages-0001:~$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.18.0 ? ... OK (13.18.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API unreachable
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... FAIL: 14:failed to connect to all addresses. debug_error_string:{"created":"@1622187622.537605962","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3952,"referenced_errors":[{"created":"@1622187622.537604821","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
Redis version >= 5.0.0? ... Exception: Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.31.1)
Git user has default SSH configuration? ... yes
Active users: ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... no
Try fixing it:
Please enable the setting
`Use hashed storage paths for newly created and renamed projects`
in GitLab's Admin panel to avoid security issues and ensure data integrity.
For more information see:
doc/administration/repository_storage_types.md
All projects are in hashed storage? ... Exception: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
gitlab server config
## GitLab URL
external_url 'https://gitlab.domain.tld'
gitlab_rails['time_zone'] = 'Europe/Berlin'
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.domain.tld.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.domain.tld.key"
### OmniAuth Settings
###! Docs: https://docs.gitlab.com/ee/integration/omniauth.html
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml','azure_activedirectory_v2','gitlab']
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_user'] = ['saml']
gitlab_rails['omniauth_allow_bypass_two_factor'] = ['azure_activedirectory_v2']
gitlab_rails['omniauth_providers'] = [
{
"name" => "azure_activedirectory_v2",
"args" => {
"client_id" => "id",
"client_secret" => "secred",
"tenant_id" => "tenandid"
}
}
]
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.domain.de"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "user@mail.domain.de"
gitlab_rails['smtp_password'] = "password"
gitlab_rails['smtp_domain'] = "domain.de"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
gitlab_rails['gitlab_email_from'] = 'gitlab@domain.de'
gitlab_rails['gitlab_email_reply_to'] = 'gitlab@domain.de'
letsencrypt['enable'] = false # GitLab 10.5 and 10.6 require this option
gitlab_rails['lfs_enabled'] = true
registry_external_url 'https://gitlab.domain.tld:5050'
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.domain.tld.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.domain.tld.key"
gitlab_pages['access_control'] = true
pages_external_url "https://pages.domain.tld"
gitlab_rails['backup_upload_connection'] = {
'provider' => 'Google',
'google_storage_access_key_id' => 'key',
'google_storage_secret_access_key' => 'accesskey' ,
'path_style' => true
}
gitlab_rails['backup_upload_remote_directory'] = 'domain'
gitlab_rails['backup_keep_time'] = '1036800'
# Extra configuration
pages_nginx['enable'] = false # boolean needs to be lowercase for gitlab to accept config
gitlab_pages['enable'] = false # boolean needs to be lowercase for gitlab to accept config
gitlab_rails['pages_path'] = '/var/opt/gitlab/gitlab-rails/shared/pages'
gitlab pages server config
roles ['pages_role']
pages_external_url "https://pages.domain.tld"
gitlab_pages['gitlab_server'] = 'https://gitlab.domain.tld'
pages_nginx['redirect_http_to_https'] = true
pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages.domain.tld.crt"
pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages.domain.tld.key"
letsencrypt['enable'] = false # GitLab 10.5 and 10.6 require this option
gitlab_pages['access_control'] = true
Possible fixes
Maybe we´ve overlooked something. Is it even possible to use custom domains with SSL/TLS on seperate pages server?