Mattermost "Token request failed" with CloudFlare (or self-signed) SSL Origin CA certificates
Summary
When attempting to login to Mattermost via oAuth from GitLab I receive the following error in my mattermost log:
[2017/04/01 11:28:37 UTC] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=1ta8ggcib7d6przadu7u36rgho uid= ip=71.63.208.154, 108.162.246.45 Token request failed [details: Post https://gitlab.dynamictivity.com/oauth/token: x509: certificate signed by unknown authority]
Steps to reproduce
- Follow the steps to generate an Origin CA here: https://support.cloudflare.com/hc/en-us/articles/115000479507
- Install the SSL certificates
- CloudFlare General Guides: https://support.cloudflare.com/hc/en-us/articles/218408028-How-to-install-an-Origin-CA-Certificate-Other-
- CloudFlare Root Certificates: https://support.cloudflare.com/hc/en-us/articles/218689638-What-are-the-root-certificate-authorities-CAs-used-with-Cloudflare-Origin-CA-
- GitLab Docker Omnibus:
- GitLab Omnibus:
What is the current bug behavior?
Unable to authenticate due to untrusted certificate and there is no way to add certificate to be trusted inside gitlab-docker. I tried everything, followed tens of Google search results regarding this issue with GitLab/Mattermost
What is the expected correct behavior?
The ability to authenticate using oAuth and CloudFlare Origin CA certificates or self-signed certificates
Output of checks
Results of GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.10.2
Sidekiq Version:4.2.7
GitLab information
Version: 9.0.2
Revision: 3e59fd2
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://gitlab.dynamictivity.com
HTTP Clone URL: https://gitlab.dynamictivity.com/some-group/some-project.git
SSH Clone URL: [git@git.dynamictivity.com:2222]:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 5.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)
Checking GitLab Shell ...
GitLab Shell version >= 5.0.0 ? ... OK (5.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... no
User id for git: 998. Groupd id for git: 998
Try fixing it:
sudo chown -R git:git /var/opt/gitlab/git-data/repositories
For more information see:
doc/install/installation.md in section "GitLab Shell"
Please fix the error above and rerun the checks.
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/2 ... ok
4/5 ... ok
5/6 ... ok
5/7 ... ok
5/8 ... ok
6/9 ... ok
5/10 ... ok
3/12 ... ok
7/13 ... ok
7/14 ... ok
4/15 ... ok
2/17 ... ok
2/18 ... ok
2/19 ... ok
2/20 ... ok
15/22 ... ok
2/23 ... repository is empty
11/24 ... repository is empty
2/26 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
4/2 ... yes
4/5 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
6/9 ... yes
5/10 ... yes
3/12 ... yes
7/13 ... yes
7/14 ... yes
4/15 ... yes
2/17 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
15/22 ... yes
2/23 ... yes
11/24 ... yes
2/26 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 13
Checking GitLab ... Finished
Possible fixes
Adding this to gitlab.rb
: mattermost['service_enable_insecure_outgoing_connections'] = true
Nowhere in the documentation did I find this, I ended up seeing it after hours of work, lost time and frustration on this page as the last post: http://forum.mattermost.org/t/gitlab-mattermost-token-request-failed/824/7
At the very least, this should be very clearly documented because I am seeing tons of people experiencing the same or similar issues. I finally got it working by reconfiguring GitLab with that above Mattermost configuration override.
Documentation Suggestion
Adding mattermost['service_enable_insecure_outgoing_connections'] = true
to gitlab.rb
should be added to this page: https://docs.gitlab.com/omnibus/gitlab-mattermost/README.html for people experiencing issues with self-signed certificates or external certificate chains, such as CloudFlare Origin CA.