Upgrading to 8.11.4 disabled mattermost webhooks and is causing 404 and 50x errors
We upgraded our gitlab instance from 8.10.x (I don't remember the specific patch release, probably .5 or .6) to 8.11.4. The upgrade went well and everything appeared fine. However, our Mattermost instance is now misbehaving. We're using the bundled mattermost install and are experiencing the following:
- Webhook settings (incomming, outgoing, and slash commands) were all reset to default (disabled). This broke many integrations that we use. Upon re-enabling these at the server level, webhooks began to function again (It seems that they were not removed, just disabled)
- Seemingly randomly, our browser themes are being reset to the default theme. Reloading the page resolves this issue
- Seemingly randomly, we're getting
We received an unexpected status code from the server. (404)
. The following is in the developer console of my browser:Failed to load resource: the server responded with a status of 404 ()
bundle-3.2.0.js:29 method=getStatuses status=404 statusCode=undefined err=Error: Unsuccessful HTTP response
- The log file in the mattermost system console is filled with these errors (one every 30-60 seconds)
[2016/09/08 18:41:08 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:41:29 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:41:38 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:41:59 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:42:08 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:42:29 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:42:38 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:42:59 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:43:08 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:43:29 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:43:38 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:43:59 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:44:08 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:44:29 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:44:38 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:44:59 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:45:08 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:45:23 EDT] [EROR] /hooks/5q15hcs1pff5xnurcq49i7gzdy:incomingWebhook code=501 rid=y9ne1jtsqjnnbyzh41169q4wia uid= ip=10.10.4.198 Incoming webhooks have been disabled by the system admin. [details: ]
[2016/09/08 18:45:29 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:45:38 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:45:59 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:46:08 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:46:29 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:46:38 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
[2016/09/08 18:46:41 EDT] [EROR] ReadFile: api.file.read_file.reading_local.app_error, open /var/opt/gitlab/mattermost/data/teams/njj9muxbcjyzzkecekk35rtzhc/channels/mtzm8imx1pnruguydgqzqa5irw/users/y8ne18h7c3ydujjago6et5wkgy/gdhqpsbbebgjtnxkiooscxb83h/Image Pasted at 2016-8-19 10-26.png: no such file or directory
[2016/09/08 18:46:41 EDT] [EROR] ReadFile: api.file.read_file.reading_local.app_error, open /var/opt/gitlab/mattermost/data/teams/njj9muxbcjyzzkecekk35rtzhc/channels/mtzm8imx1pnruguydgqzqa5irw/users/y8ne18h7c3ydujjago6et5wkgy/gdhqpsbbebgjtnxkiooscxb83h/Image Pasted at 2016-8-19 10-26_thumb.jpg: no such file or directory
[2016/09/08 18:46:41 EDT] [EROR] /api/v3/teams/njj9muxbcjyzzkecekk35rtzhc/files/get/mtzm8imx1pnruguydgqzqa5irw/y8ne18h7c3ydujjago6et5wkgy/gdhqpsbbebgjtnxkiooscxb83h/Image Pasted at 2016-8-19 10-26_thumb.jpg:writeFileResponse code=404 rid=gi34exrwbifq7nbwp71kmnwemc uid=y8ne18h7c3ydujjago6et5wkgy ip=10.34.6.105 Could not find file. [details: path=teams/njj9muxbcjyzzkecekk35rtzhc/channels/mtzm8imx1pnruguydgqzqa5irw/users/y8ne18h7c3ydujjago6et5wkgy/gdhqpsbbebgjtnxkiooscxb83h/Image Pasted at 2016-8-19 10-26_thumb.jpg]
[2016/09/08 18:46:59 EDT] [EROR] /api/v3/users/status: code=404 ip=10.34.11.86
These problems persists after restarting the stack (gitlab-ctl restart
)
Mattermost is configured as follows:
# Mattermost Config
mattermost_external_url 'https://mattermost.mydomain.net'
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mattermost.mydomain.net.crt"
mattermost_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mattermost.mydomain.net.key"
mattermost['service_use_ssl'] = true
mattermost['email_enable_sign_in_with_email'] = false
mattermost['email_enable_sign_up_with_email'] = false
mattermost['email_send_email_notifications'] = true
mattermost['email_smtp_server'] = "postoffice0.mydomain.net"
mattermost['email_smtp_port'] = "25"
mattermost['email_connection_security'] = nil
mattermost['email_feedback_name'] = "GitLab Mattermost"
mattermost['email_feedback_email'] = "mattermost@gitlab.mydomain.net"